A Software Engineer Reacts To Star Citizen Server Meshing & Replication Layer - CitizenCon 2953

By popular demand (thanks, both of you ;) I share some thoughts from a software engineering perspective about the Replication Layer and Server Meshing section of the StarEngine talk from Star Citizen's CitizenCon 2953 this October 2023.
2023 CitizenCon presentation on replication layer & server meshing:
• CitizenCon 2953: Shapi...
2021 CitizenCon presentation on persistent entity streaming: • CitizenCon 2951: Serve...
Jake Beal paper on "What the Assassin’s Guild Taught Me About Distributed Computing" from MIT: jakebeal.github.io/Publicatio...
Overwatch Developer Update | Let's Talk Netcode: • Developer Update | Let...
Zachery Johnson talks about Blizzard's Diablo II: Resurrected database architecture on the Critical Point War Stories podcast:
• HE LOCKED PLAYERS OUT ...
My main channel: @critical-point
My personal site/contact info: free-dissociation.com/contact

[@CSKefka]

Apparently when the guy who made this first showed Chris Roberts, he was told that if we wasn't a coward he would showcase it live. Well done.

[@zf4120]

"It should not be possible" is usually followed by "How did they do that?"

[@gordonhowett7529]

Dude this was exactly the kind of professional opinion I was hoping to get from this video. Great work, great insight. You got another subscriber here!

[@tenthlegionstudios1343]

Game dev system design is nuts. Low latency multiplayer games where you are trying to sync everything in real time is a hard problem. Thanks for the content!

[@RealKanashii]

IT crew use to say: there's no real time, conceptually and technically. So yes, real time is always a problem (and that's why they use the "ticks"). :(

[@edward3190]

it's impossible. SC will fail no doubts. The reason MMO games use regional servers is that the speed of light is finite. Light travel from Asia to America takes more than 100ms, which is enough to destroy competitive gameplay. SC choose to ignore physics is beyond insanity.

[@claiyc]

@@edward3190 misinformation my friend. The goal of sm is not to overcome latency problems in terms of "letting people that are on other sides of the world play together", but rather to just distribute load by splitting simulation of a gameworld into different processes that can run on different machines. Everyone is well aware that latency will always be a problem, that's why it's highly expected (by devs aswell as community) that there will always be regions (NA, EU, ASIA, ...) in SC. Having ONE global shard is an existing idea, but no one guarantees that smth like that will ever exist (perhaps there are options to achieve something with instancing and concepts of "consistency over time" but that's it from my pov - no real time gameplay interactions between two clients that are on other sides of the world)

[@nicodemiusoverschnaps4759]

distance is approx. 10.000 kms at 299 792 km/s... more than 100 ms.... Suuuuure. ROFL.

[@edward3190]

@@nicodemiusoverschnaps4759 just shut it if you don't understand real life. light doesn't travel that fast on Earth, and 10000km... wtf?

[@sc_cintara]

The distributed lock and all that is cool, but the really big job was that all state had to be externalised from the game code and copied into the replication layer. That is *all state*, because a server has to be able to take over in-flight or take over from a server that crashed at any moment. So essentially, server meshing changes the whole code base, all code in the game servers. I saw another hint in some other presentation about how things work: They said explicitly that the entity graph is distinct from the replication layer. I agree that the entity graph might well be Neo4J, but I think the replication layer is something else, probably in-memory. Also, in last year's Chairman letter, CR mentioned that the replication layer is horizontally scalable and redundant, so the replication layer does need to handle distributed lock and replication in a cluster. BTW, I have founded dev companies so I can guess why CR cried when it worked: CR sent off this team to build server meshing many years ago knowing full well what the job would require (he is a good dev on his own merits; he wrote most of Wing Commander himself). Starting a project the size of server meshing is a "bet the farm" project. If the dev team had not been able to get server meshing working, CIG would quite likely be over. So CR had to trust that the dev team were able to do it, entrusting them with the future of CIG and let them forge on ahead through all the delays and the failure of iCache. That is a nerve wracking position to be in when you are responsible for 1200 people's livelihood!

[@JohnMcclaned]

could be memgraph

[@OrniasDMF]

and when people are constantly calling your game a scam

[@coolkid15105ify]

Thanks for your insight and context :)

[@Hax187]

50:46, the bullets are ray casts which is incredible to me. There was a interview with one of the developers and he said that raycasts are working but projectiles are not working atm

[@Crispy_Steak]

sorry, do you happen to know anything more about this interview? Whose interview or a link?

[@originalgameronline3457]

My man had a Stargate reference in his video. That always deserves a like. o7

[@dmacpher]

And Enders Game with the Ainsble

[@free-dissociation]

Originally the ansible was Le Guin, from whom Card borrowed it, but yes I am a big ol' nerd and have been forever ;)

[@dmacpher]

@@free-dissociation Ursula! Didn’t know lol

[@thenecrolept]

Don’t have much to say beyond, very interesting commentary, very cool insights. Definitely subbing in case you do more similar content in the future. Or if you get that interview!

[@artuno1207]

"This is not something a AAA publisher would have put up with" this is why Ive believed in this project. This why ive been patient. Theyre doing soemthing phenomenal. I want them to succeed so that all the haters of this project and those who call it a scam have an amazing game to play soon(TM).

[@sc_cintara]

Yes, the funding model is the real secret sauce of Star Citizen. No financer would approve of waiting this long to start selling the game if it had been done in the traditional way. All that money sitting in CIG would be money not giving of interest, so the expected profit requirement for the game would grow year on year until they become untenable. At that point the game would have been forced to publish to recoup as much of the loss as possible. It is the main reason why Forbes hates Star Citizen; the funding model entirely cuts out the financer.

[@r2artCH]

I think the game already good with all the current features. Only one system but there's so many things to do. There is only two things that bother me with the current state, and that is the bugs and the NPC AI. If they fix those two, I can say the game already complete even with only one system. With this player already able to play the game, while they working on expanding to a new system and new vehicle internally until it's ready, stable and minimal bugs they can update it then continue expanding internally until ready. Just keep doing it like that, it should be good.

[@sc_cintara]

@@r2artCH Unfortunately, having each release mostly bug-free will slow down development quite dramatically. Think of it as building a something like a chest of drawers. You build it with some group of drawers to begin with, polish it, make sure all the nails are tightly nailed in, etc, and give it to the customer. Then you want to add new drawers. To do that you have to break apart much of what you already built to fit the new drawers. Then you have to re-polish it, fix it up, etc to make it look nice again. This is the problem with releasing code under development: If you want it to be bug free and polished, you have to redo the bug-fixing and polish every time you release and that slows down development dramatically. So CIG are doing minimal polish and accepting quite a lot of bugs, especially in systems that are planned to be replaced, to avoid getting all resources stuck in bug-fixing and polish.

[@Dovahkiin53]

Thanks, well explained and thoughtful.

[@nekomancer4641]

Very unique take on the topic. Good to hear from ya

[@jasiekthejester7764]

Awesome dose of information! Thanks!

[@Billy-bc8pk]

At 45:40... they're handling it via Entity Zoning through their SSOCS infrastructure, which is really impressive tech. Interestingly enough they were testing the server mesh via Entity Zoning back in June with limited Evocati participation just to ensure that it actually worked. The instructions were to have 200 players move from one specified object container to the next (I believe they started at Everus Harbor and then had to quantum to another station/planet designated as a separate zone) to see how well seamless transfer authority worked.

[@juz3r1]

Such a test is currently underway at EVO!

[@artlife9563]

Well this example is a very small room. The key is that currently, one server has 5 planets and several moons. Imagine if one server only had to account for one planet. The data strain will be greatly reduced. The transfer latency between planets or moons wont have to be instant as shown here, because planets are so far apart.

[@jafogx]

Yeah this is something that I wish they had made more clear. The way I’m picturing this is in the game there is no reason to have three different servers handle three different rooms in the same building, this was just a scale model to show the technology. Ideally, a server would take care of a large area (as you say, could be a whole planet, a station, etc) and then we wouldn’t notice any lag because we’re in quantum when it happens. That would also mean the example he mentions where you’re shooting something across three servers is infeasible because of scale. They don’t mention this specifically on the conference though, so it’s kinda up to our own interpretation. They could’ve said “these rooms would represent different planets, or a 1/4 section of a planet” or something to give a sense of scale of how it would actually look in the final game.

[@uttrik]

Though the end goal is to have the mesh dynamically scale based on the number of players in a region. If a thousand players show up on microtech, multiple servers are going to have to take care of a single planet. And if they all show up at new babbage, there may very well be situations where rooms are divided up into separate servers. Maybe the designers will make it so server boundaries will never have situations where you can shoot through multiple servers when indoors.

[@free-dissociation]

Because they demo shooting across server boundaries I kind of infer that they actually intend for that to be a thing in the medium term.

[@_R8ze_]

​@uttrik this person gets it!

[@grygaming5519]

@@uttrik Oddly this is something MMO's have to deal with and are unable to deal with. Case in point just look at Final Fantasy XIV Endwalker's Login issues, 5000 players all trying to log in at the same time the servers were unable to handle it. Now if this teach was in its growth/maturity stage something like having login issues for MMO's or dealing with crowded areas would just be a thing of the past because the system would handle the issues without a problem.

[@Alundil_55]

Thank you and well done

[@sharxbyte]

this is an incredibly intriguing as an engineer and amateur programmer! Also thanks for introducing me to the term "Nerd Sniping". Love Munroe

[@AlbandAquino]

50:45 To answer your question: Yes, they are. But instead of having a 3D mesh to represent the object it's treated more like a physicalized particule (GPU bound). So you won't see a mesh around those. But they are physicalized. 59:19 That sound like something you went through 🤣 I'm a DevOps engineer, specialized in IAC. I felt that moment... 😅

[@toecutter9372]

what do the blue and yellow colors mean for your pic?

[@AlbandAquino]

A sea of sunflowers under a blue sky. But realistically, the Ukrainian flag. @@toecutter9372

[@MaticTheProto]

@@toecutter9372something good

[@free-dissociation]

> That sound like something you went through 🤣 I have stories. So many stories ;)

[@Blu3Souls]

I do whole-heartedly believe that the dog-leg is there because they wanted to show the stream-out mechanic. If they let us shoot from one region into a different region across a third region then the two outer regions will probably not stream each other out. How that works with dynamic server meshing where you can't know what zone has "view" into other zones will have to be decided (I'd bet on distance between the zone edges)

[@dawnphantom2439]

I think the idea is that at scale, one server is Microtech, the other server is Arccorp, and the third server is Crusader. The distances between server zones is so vast, that this won't even matter to begin with and the idea is you'd only ever need to interact with things in the nearest server over anyways, and never need to reach through the next server to reach the third.

[@bambooza1]

I agree the dog legs purpose was to allow them to showcase what happens when game objects go out of view. One thing I think keeps getting overlooked is that from the engines point of view all movable objects have an origin point in the parents reference view. So when you move an object you only need to update the objects origin point, and not every point that makes up the mesh. The boundary box for each server mesh then can be a simple coordinate check if that point has crossed the boundary and can be handed off ownership. This prevents a lot of calculations and the issue of portal slices of game objects. Then the players graphics card can do the heavy lifting of matrix calculations on the objects mesh to camera view. It is also more than likely the same values used to reference in the replication layer and storage. You just need to know the object ID, the object origin point in parent coord values, parent object a list of child objects, and list of mod values. I know they said it's being stored in a graph database but given objects can't be inside themselves there is no potential for circular paths and so it's just a tree problem.

[@sodiufas]

this show is only meant to early wh, oh baker reassurence

[@ThatOneCakeGuy]

@@dawnphantom2439 The end goal is much smaller, but in the demo they kept it limited to the 2-3 fields which is just a place holder. To start out it's going to be one system per node, then likely planet, then probably regions of planets, POIs, etc. down and down as the tech evolves and population increases. The end goal is for it to have "as many as needed" to handle the population of an area. Ie if one area is more popular it may have several smaller zones with each master having a further view where as a low pop area might only have one zone covering a larger area because less people go there.

[@gruller65offroad96]

Remember thease zones will be significantly bigger , think station , quarter solar system size to this issue isn’t really a big deal , only pes is the biggest thing

[@JohnZingTTV]

this video convinced me to install the free 14 day epo install and try this game even with my poopy 4gb 1050 gtx ti gfx card keep up the content i think the 20k views show there is a high demand for this content and the format it was very real i look forward to see whatelse you will cover as far as other games are concerned. well next convention if yougain alot of traction on youtube mabey they will pay all exspenses so you can go taere nextime to citcon right? :D

[@consumablecorner150]

Firs time watcher. "Perfect is the enemy of the good." Never heard that before! "..the truth will set you free." Definitely an empowering statement. You're a very soft-spoken bro. In fact, so much so that I could hardly hear you xD (edit: at least in the beginning)

[@budthecyborg4575]

Very insightful about the handoff lag. The best way I can think of hiding handoff lag right now would be to make sure server areas are all sized well beyond the practically implications of player input, relative to the size of the game the range of your bullets is nothing. Forget about three servers in one hallway, the smallest server zone in v1.0 of server meshing will probably be an entire planet/moon, areas separated by millions of Km. I would expect handoffs to only take place in the travel between planets at which point the likelihood of players engaging in combat on the border between servers would be effectively zero. As-is we're limited to about 40 people in each instance of the entire solar system, giving one server to each separated POI in the system would allow an absolute maximum of about 1,200 people (there are around 30 individual points of interest in the Stanton system). Of course the total would still be limited by the single highest traffic point in the game, but I would expect at least an order of magnitude more players active in the system at once, once server meshing is in Star Citizen will finally feel like a real MMO.

[@kylemelder5118]

Current instance pop is a flex number of 100. They have had 200 player servers.

[@budthecyborg4575]

@@kylemelder5118 Excellent, my only info was from a gameplay video and I assumed he was correct. If CIG servers are good for 100 players per server then Server Meshing is going to make Star Citizen really pop off.

[@mobiuscoreindustries]

​@@budthecyborg4575 the thing is we know somewhat they they can already as people have stress tested the servers by bringing all 200 people of a server into a single ship. And barring restricting people to walk (so people would not die off running into each other and colliding) the servers were actually running better than normal in this scenario. It's because right now the cause of server load is due to all the players being spread out everywhere, meaning the server has to load and account for basically every major landing zone and most points of interests at the same time. All those interactions stacking on each other until something bad happens. Essentially, CIG's streaming tech can't be utilized to its fullest server side as long as servers functionally will have to keep most areas loaded.

[@free-dissociation]

Yeah, it's certainly possible to set up server meshing so that it's very rare for players to cross servers and thus hide server leg. Because they demo'd interaction across servers, I infer that they actually intend for that to work and be a big part of how the game works at least in the medium term.

[@budthecyborg4575]

@@free-dissociation My bet is they need localized server meshing to work before we can have a playable Bengal. Players have speculated for years about a single Idris meshed across multiple servers but that would never actually be necessary just for an Idris, the Bengal on the other hand is likely to rely on this system.

[@TheKingoftheriff]

"lightning bolt, lightning bolt, lightning bolt" Fucking subscribed.

[@Sarsour_]

Awesome content!

[@MrJabbaStyle]

this is the stuff that i don't understand at all but still understand enough to be highly impressed!

[@JC-Alan]

Great video brotha. Dont worry about background noise :) not much you can do abiut a hungry roommate. It's good to watch this after attending CitCon with my buds that arent in tech. Im neither a game dev or a regular developer, but as a systems admin I have some familiarity with data systems. Your comment about localizing the inter-server communication to something like a kubernetes cluster is spot on, if i had to guess. This falls in line with their initial idea of geographic shards (e.g. a shard for NA, SA, Europe, and Asia, etc), but that's gonna be a huge undertaking if they want the solution in finality to have everyone on a single shard. The laws of physics work against you there, so im very curious to see what directions they take with minimizing the latency on handoffs.

[@sc_cintara]

We are already able to play in distant regions if we like (you can select region manually when joining the game) and it works, though admittedly 0.5s ping is a bit "bouncy" 🙂. Nevertheless, unless you are doing combat, it is usually fine. An issue with shards is that they are so isolated from each other. You will never meet people who play on other shards. This is an issue for large global orgs with people wanting to play in their local shard, yet hang out with org members from other parts of the world. One solution could be to take advantage of the Amazon infrastructure and high performance networks between their DCs to set up a "follow the sun"-shard. So, if you log in to the "follow the sun"-shard, you will be in the region that is currently in it's prime time, but as the 24 hours move on, your server gets migrated to new DCs around the globe. There might also be local shards, for those that always want perfect ping, but if you are OK with high ping during off-peak time of day, you could make your home in the "follow the sun"-shard.

[@IronPhysik]

I have a feeling that you may have misunderstood the job of the replication layer in all of this the way I myself understand it is that the RL is there to replicate objects for other servers on other zones so that each server can do the physics calculations for its own zone of objects, and it just displays the results of what the others did so its "replicates" objects for the player, and thats how you can shoot across boundries.

[@ITHYANDEL]

I think it is not using a physicalized bullet, but a time constrained spline, the spline would then be subdivided for each server through the replication layer, which keep the full spline.

[@bambooza1]

Ya they said they went back to hit scanning from physicalized bullets.

[@Whoadayson]

Cool video, Thx! CIG is doing big things, but I still have my skepticism that they are as far as they want us to think! Sidenote: dont worry about background noise. Its narely noticeable!

[@robmulally]

So interesting you mentioned the interaction at a distance, this was solved by another massive multiplayer game by making ordinanace that could possibly traverse withotu the player like large bombs or whatever, ownership was passed over to the server, and thereforce would just act in CIG case like a server owned object passing through each zone. This also allows objects that impact large numbers of people independant of the player, this is also basicalyl how i assume a player leavin their ship in an are would be handled if player despawned in combat etc, so I think this will be built in by default.

[@cryingsurrogate]

A minecraft mod called "Create" pulled off a similar thing with trains - travelling through unloaded world chunks. Since the rails are curves, it can follow them regardless of the chunk state. I'm guessing this pre-calculated curve approach would be the type of implementation for this game, since bullet distances are finite.

[@NoNaNelson69]

transition is most likely when its almost done- if you look at armistice zones, its basically the same how to server detects once ur in or out you enter when you are fully in and leave when fully out

[@honordevs]

What a lot of laymens don't understand is that the entity view is actually just the graph database. It's an advantage of using one, like neo4j, that you can easily present the information. Most of my friends thought it was a custom tool, (which is definitely is modified, but i digress) that they made for debugging because they had never used a graphdb.

[@ParaquatSC]

A note that each game client around @50:00 shows how many entities are tracking and the act of shooting doesn't appear to affect that counter at all. So bullets would appear to not be entities.

[@vorpalrobot]

The projectile system is different than the one that handles objects. I think they might have tried to physicalize bullets etc early on, but I remember them excitedly showing off the new projectile manager with a big fleet battle with thousands of shots in the air.

[@ParaquatSC]

@@vorpalrobot definitely. Also tracking projectiles moving at 1000mps+ sounds very complex once you add latency!

[@ParaquatSC]

I also think without a doubt that the server zone will be a concentric area within another server. That way there will only ever be one or two "borders". So Stanton as a massive playzone, then a zone for each planetary system, then perhaps a zone for the main city, just as an example.

[@richardmoore4194]

Is managing these transactions in a node DB compared with RDBMS so much harder , or is it that we have 40 years experience and mature methods and patterns associated with managing RDMS. I am so old I remember switching from PICK to Dbase III early ORACLE had all sorts of undocumented features. Thanks for doing this BTW , i much prefer this level of insight with house mates making late night snacks than glossy videos that gloss over the issues 🎉

[@eltreum1]

Speaking from experience of converting an RDMS to GDMS It's a little of both. SQL has the advantage in write speed for realtime transactions especially if you don't have varchars fields and fully rigid rows of reasonable size. A true GDB like Neo4j destroys SQL in search/read speed regardless of database size the moment in SQL you would have to use the Join or Union keyword. The way you model and query the data are very different mindsets between the two when designing them and your apps.

[@richardmoore4194]

@@eltreum1 So RDMS are faster for ACID transactions, but GDMS are better at reading records and significantly equally fast irrespective of size of the DB. So will some sort of caching / queuing system be needed. Providing it was a real working demo - the focus will be to make it stable - then make it fast.

[@eltreum1]

@@richardmoore4194 A couple years ago they implemented what they call icache as a caching layer that is the foundation to the replication layer. It allowed servers to crash and us not lose ship and cargo every time if our cargo made it to cache before the server crashed. GDB isn't drastically slower on writes, but you do have to plan your workloads if you want consistent speed per transaction. EvE online did a similar thing to address the DB load/response time. The entire DB was replicated in a mirrored pair of RAMSAN cache arrays connected to the server cluster via Fiberchannel backplane. The RAMSAN also had read replicas on SAS storage arrays for backups and read queries that did not need real time priority. NVME and new gen RAM storage arrays now eliminate disk seek time mostly if you need it. SC is a little different and a game that will play itself. There is a universe simulator that is just a machine learning system that calculates the macro economy and state of the universe stats and designated target metrics and employs NPCs to balance things as players affect systems. That is mirrored to/from the replication layer and other backend services that are like APIs in a microservice system with the replication layer as a data broker. The game servers will be assigned authority over parts of the universe and any players or NPC that travel there so the live state is ran with distributive computing. NPCs the players can't see are running as microservice agents on other server pools streamed off the gameplay servers interacting with the universe system at the speed of a player for travel/work through APIs to interact with the replication layer. When NPCs are in sight of a player they are streamed back onto the gameplay servers so you can see and interact with them. That is part of the plan to get capacity and scale is offload the gameplay server anything a player is not going to see so the universe keeps moving. Crimes and cargo NPCs keep working to modulate the security and economy to a target in response to players messing with things. Sorry for long post you should watch this vid where their engineer shows the stuff in action. kzbin.info/www/bejne/aJ7YeIqqjZ2Ybck

[@criticalchai]

The big worry i had was latency. But then I thought if they cluster the various servers running the replication layer and shards (assuming shards would take on the role of each hosting a couple of containers) that moving from one shard to the next shard should be pretty fast and have pretty low latency between each other even if they have high latency between the servers and us. they may have to figure out clusters and probably if enough people showed up in one spot it would leak outside of the clusters to find any available shard to host containers. I'm curious how well it will deal with a ship fill of loot from bunkers when it transitions between servers/shards

[@miinyoo]

Yeah that's the real pickle. How does one do that with so many children? It seems possible but when the electrons hit the wires... Like FD mentioned. Protocol is everything if it is to be playable. Simple compressible and elegant.

[@Knight3rrant]

Not really. The entity within entity within entity capability of the graph database means you just move the shi entity and the associated entities within it go along. This can work for the ship entity as easily as a container entity.

[@Nemethon]

I think it is the most important part of the whole project. :)

[@seanc6754]

I been a backer for about 4 years and I'm with you buddy i have a potentially unhealthy obsession with this game myself lol..even when i take a break from playing i watch a LOT of sc stuff lol

[@matthewfletcher4627]

The fact they are persisting entities across servers, is just a showcase of this... In reality... A server is going to swathe a HUGE span of the system... IF and the IF is pretty big here... players find themself on the cusp of a server boundry, yeah... Might be difficult... But as that could be the edge of a bubble that's millions of meter cubed.. Chances are slim... Getting to the edge of that bubble would mean you dropping out of QT at the right time

[@freelancerthe2561]

bold of you to assume theres not a lot of dead space in between, in spatial terms. One of their main goals with meshing and containers is to not manage space with nothing in it. The other half of that is to abstract out large chunks of the NPC simulation, so you don't need to spend a lot of server resources on them when there are no players to observe them. Its not a fixed number of bubbles across space, but a bunch of small roaming bubbles that merge and break off as things move around.

[@matthewfletcher4627]

@@freelancerthe2561 I don't think there will be... All that dead space will be on its own shard...

[@MaticTheProto]

@@freelancerthe2561dead space coincidentally is very easy to calculate as there is nothing there

[@mobiuscoreindustries]

Right now the edges are going to be fixed. Essentially they will delimit areas of big traffic and assign servers to each. But the goal is dynamic server meshing, meaning server authority changes based on the actual load demand. So that if a big chunk of load moves from one stellar area to another then servers will gradually cluster up in that area to divide the load. Potentially up to the divide happening within visual contact. Everything else that does not need to be seen is streamed out, and handed to the background simulation, which is an entirely separate thing that feeds to and from the in game simulation.

[@strubbleler]

shiny man is SHINY ! love to see new people get into the game, and be amazed just as i was, even though im not a developer! love the yellow wall by the way!

[@nagasadhu3286]

FD's a 2014 backer ;)

[@Fluke2SS]

Overwatch is a bad fucking example, because despite the fact Blizzard said the principle of "Favor the shooter" is in place, what they don't tell you is the server tick rate is 60 or 64hz. So if you are running 120 FPS constant and you take a shot, you miss because the server (aka Replication Layer) misses 1/2 of the information the client sees. They recently changed Sombra's translocator ability which is directly effected by server tick rates. You can do a 180 degree turn throw the translocator, and the "Replication Layer" only sees you turning 90 degrees and as a result your translocator hits the wall next to you instead of where you were actually aiming on the client side. This is the same reason people claim to be behind a wall or corner and they die in a different location than what they see on their client. TL:DR Overwatch needs a 120hz+ server tick rate to make the game playable for most clients considering how extremely low the hardware requirement to play Overwatch actually is. CIG might be able to get away with this issue at 60hz tick rate because the game requires beast hardware to run. And it is not even optimized yet.

[@sevilnatas]

I wonder what type of message bus tech they are using. It's got to be fast and I wonder if it is a guaranteed delivery system or they have room for lost messages.

[@kiblams]

The room used in this was is just for demonstration. I doubt they will use small areas per server in end use. They are more likely to have a city as one server and maybe the inside of a large ship being one server. So movement between servers wouldn't happen as often as you are assuming.

[@BrandonFerrell]

They have stated that the meshing is intended to be dynamic based on load in the future. You may be correct that the initial implementation might not have this issue, but they have set themselves up to need to tackle this issue in the future. An example they have given in the past is a single ship, like an Idris, having multiple dynamic meshes. "With Dynamic Server Meshing, it’s possible that large ships such as a Javelin could have their own dedicated server assigned to run the authoritative simulation for that ship and everything on it. However, we’re trying to avoid having inflexible rules about how entities get assigned to processing resources, so that might not always be the case. It comes down to efficiency in terms of both processing speed and server costs. If we had a hard rule that each Javelin and everything in it gets its own server, then it wouldn’t be very cost-efficient when a Javelin only has a handful of players on it. The same rule also wouldn’t be efficient in terms of server processing speed if there were hundreds of players all crowded into the same Javelin, as the rule would prevent us from distributing the processing load across multiple servers." robertsspaceindustries.com/comm-link/transmission/18397-Server-Meshing-And-Persistent-Streaming-Q-A

[@Shawnsrumi]

As SC backing I appreciate your perspective…thanks for the insight

[@pale0wl]

A mounted SHURE SM7B will help with your roommate situation. You will want to set a gate though. You can gate through OBS but having a gate on the audio interface is nicer for monitoring

[@sevilnatas]

When he had the graph representation up, I don't remember seeing a bunch entities being spawned off the zone node, that you'd expect to see if they were physicalized entities, like other entities. Wonder how they are handling that?

[@free-dissociation]

Other commenters tell me that CIG moved from physicalized bullets back to hit scanning.

[@sevilnatas]

@@free-dissociation Ah, makes sense.

[@sc_cintara]

@@free-dissociation I don't think they are doing hit-scanning in the immediate-hit-where-you-aim-sense. I think they are doing bullets on the GPU. At the least for ship combat there is a hard requirement to follow where the shots are moving (those beautiful arcs of fire between ships), so for large scale fleet combat they still need to move bullets between zones. However, I have some vague memory that even for ground, they still trace bullets through the air, they just have some GPU trick to draw them very quickly rather than traditional geometry. BTW, this is easily testable with a sniper rifle.

[@VectorParalax]

I think you just became a Star citizen youtuber!

[@PlumHeadLJ]

51:20. I think there is no bullets, just raycasting outputs from player entity with saving the angle (vector) when leaving one zone and receiving it by another zone

[@TheKingoftheriff]

You also need a timestamp and an owner, bc the player isn't on the second server.

[@marclamy364]

Players, just like servers, have a streaming container around them. It is likely that servers only need to see around them to facilitate simulation and transitions, but players have variable needs. For instance, you'd expect a player flying at 1200m/s to need a much bigger container to stream entities early enough to avoid popping. So to answer your question, if all 3 servers in this demo were in a straight line (purple, green, red) and bullets were physicalized, you could imagine a player in server purple shooting at a player in server red. The server purple would manage the physicalized bullet for a while, then server green and then red, potentially handling the collision with another player there. While the shooting player needs to see the result of their shot from server purple to red, server purple doesn't care. So they'd have very different streaming containers.

[@HitmannDDD]

We saw a demonstration that carved up the game space by 3d spacial boundaries because it's easier to visualize in a demo. However, given what you know about the tech, what additional technical challenges do see regarding removal of the spacial boundries for servers? The use case being that multiple servers could then handle a single 3d gamespace, thus potentially increasing the max player count in close proximity.

[@matthewvelazquez2013]

So, uh, The inference I'm making at minute 46 Is that the same concepts behind A.I. frame generation are being applied to entities transitioning from one zone to another? That's why zone adjacency must be so important? The line of demarcation between zones Constitutes a 'generated frame.' Based on the mock up on the screen, A true frame could be symbolized As the purple zone. The generated frame could be symbolized as the green zone. And the following true frame could be symbolized as The red zone. Scale this down And compress the operation inside a single line of demarcation Between adjacent zones, right?

[@surject]

There are a few glitches in the demo, for example the 'box' in the middle left in the green zone (bottom right view) moves at 58:34 (player in the first person view shoots it) but it does not move in the fpv nor the other 2 views.. and then it suddenly re-appears again in the bottom right view at 1:00:31 - hmm?

[@RanmaKei]

I see that too and may be a bug. They did just put this all together as the first iteration of server meshing so I'd imagine they need to tweak the code to get it perfectly in sync with all the different layers of the simulation. I'd imagine you would need to do some complex checks and those are done on the server side as opposed to the client. These checks will likely be involved with latency between servers as well. It could also improve with time as they work out the tech. It's not so much about running into issues with simulation as it is probably more about how you handle those issues since you will have latency, memory pressure, CPU throttle, crashes, etc. that you have to account for in code at replication layer and simulation layer. Will just have to wait for the next iteration of the tech and see how dialed in it gets. As it stands its very impressive that they have it working to this degree.

[@free-dissociation]

ooh hmm. thanks for pointing those out. I'll have to go over this and see if I think it tells me anything more about there implementation.

[@doubledigital_]

been faking levels for ages because of memory only recently are we able to do open worlds that we dont need to cheat :D fps games like quake we use to hide parts of the level and you would only see it when you moved into the box show here.. the interesitn part is they have linked the box ( container ) to the server its self.. i aint ever seen it done this way before and i been doing game shit for 25 years give or take. impressive shit.. this in fact impressed me more than anything.

[@joshoowa]

Was this live? I'd be interested to watch the show live with you on twitch or youtube or something in the future.

[@thePrisoner1000]

Yes, it was done live at the event.

[@pylotlight]

@@thePrisoner1000 not what he was asking.....

[@80KBM]

I think it is about 10 hours total. I cant find one thing they talked about that i thought was "bad" or "missleading" And i am a bit picky.

[@Sypher474]

The presentation was live on Twitch originally then uploaded to YT, this reaction was a recorded video.

[@moonryder203]

We are all waiting for this tech to hit the live servers if they can pull this off. I have never been so addicted to a game ever in my life, this game is like every childhood dream I had of being able to do in a game and then some. 😂 This attempt to do this in a game will be historic for many in this industry.

[@gorganhorn6872]

Please tell me your uncle is Jason Issacs, because that would make perfect sense in this reality. BTW thanks for your expertise and taking the time to unpack all the details that even CIG's show cannot do. +1 sub

[@josiah9388]

It would be interesting know when a 30k Happens is the crash a single service on a server or the physical server. That would drastically change the tech. If it's a service or VM that crashes you have the replication layer on the same physical hardware. If the replication layer is on different hardware the transitions will be much more impressive. not to mention whatever tech they have for spinning up new "Servers" as needed.

[@free-dissociation]

It's just the game server service process crashing, it's not the underlying hardware. (Think how much that would have cost CIG back when we were getting 30ks every day in game if it were the hardware ;)

[@josiah9388]

@free-dissociation I agree that's how it works currently. But it may not be in full server meshing.

[@marclamy364]

You're correct about the fact that running all servers and the replication layer on the same machine, it eliminates the latency issue. That said, the latency between game servers and the replication layer, assuming running on the same data center, should be negligible. Assuming the game servers simulations run at approximately 60 ticks per second, that gives around 16ms per tick. With an average network latency of, say, 1ms within the same data center, and standard client-server synchronization techniques applied (client prediction, server reconciliation), it is very possible that no latency would be visible in this very simple demo.

[@free-dissociation]

Wellllllll. It doesn't _eliminate_ the latency issues, unfortunately-I wish it did. There's all kinds of nonsense that gets in the way. (Whatever nonsense the scheduler has decided it's getting up to that day is the one that I remember biting me personally.) But I'll certainly take same-machine latency over even same-datacenter latency.

[@LumiLupo]

you had 4 minutes of talk before the actual content of the video started, i'd recommend you to keep that a little shorter (cuts do help) so people keep watching. or you use timestamps so people can jump directly to the mainpart what means a lot of people will not see what had come before ^^

[@Crittek]

The replication layer is also moving the client using its input. Most of their idea is ripped out of Unreal Engines networking, everything is server authoritative. As a consequence, what this does and nobody is talking about is enabling very effective anti-cheat measures.

[@sxnorthrop]

Problem 1: I suspect that there's additional communication between each of the adjacent servers that allows them to notify their neighbors of any raycasts that may potentially penetrate an entity not under their authority. The question is what happens when there's latency between each server? What if the bullet has a large enough range to go through, say, 3 different servers? That would make the experience even *worse* than using a single server because you'd be tripling the latency cost. They'd definitely need to avoid placing many of these zones in wide open areas. Problem 2: If there are players that know the boundaries of each of these servers, what happens if that group all decides in a single moment to move between each one rapidly? Depending on how computationally expensive the "trade off" algorithm is a large enough group of players could potentially cause the replication server to lag or crash outright (or even cause a nasty deadlock depending on how they have things set up). Problem 3: This is a space simulator that allows you to travel hundreds of times faster than the speed of light. What happens when player 1 leaves server A and transitions to server B, but before the replication server is able to handle that and server B knows that it has player 1, player 1 is now transitioning from server B (at least it's zone), to server C, but it's still under server A's authority? Does server A still claim to have player 1 and so update the transaction to say server C instead of B? I'm sure there's something that could blow up there in the wrong circumstances, or potentially a teleportation exploit. Maybe I'm mistaken. As with all things new, something's bound to break! But G** bless them for being the first ones to try!

[@free-dissociation]

> The question is what happens when there's latency between each server? And the first rule of distributed systems is, there's *always* latency between servers. > If there are players that know the boundaries of each of these servers, what happens if that group all decides in a single moment to move between each one rapidly? This is a pretty common tactic that EVE Online corps have used to turn battles in their favor, from the launch of the game, so yeah-a big problem, and one CIG will have to address. > What happens when player 1 leaves server A and transitions to server B, but before the replication server is able to handle that and server B knows that it has player 1, player 1 is now transitioning from server B (at least it's zone), to server C, but it's still under server A's authority? Probably, the client is not allowed to initiate the transition to server C before the server acknowledges the authority transfer to server B. Hope they tested the heck out of that state machine!

[@80KBM]

Thankyou. I´ve been waiting for someone that can explain this tech in an "easy" way. A question. The scope is that this type of tech will make it possible (they say) to have (lets go a bit extreme) 5000 people on a planet surfice, and lets say 10 capital ships with 20 people inside each, and flying in low planet orbit. shooting, debris falling down and explode on the groundtroops. And all the broken ships, loot, bodies will be there for lets say"a year" for everyone to go looting, salvaging. Do you think this tech is the "answer" to make that giant type gameplay possible?

[@free-dissociation]

That's the goal, and if they're going to pull it off, something shaped like this is what will enable that. Whether they can pull it off, at a cost they can afford-we'll find out! It certainly looks like they're on the right track at least.

[@unknowncitizen9300]

Can someone give an idea, at what point the replication layer could struggle updating the entity graph with the appropriate tick rate (lets say 30)? Because for my understanding, if meshing would work as intended, this would become the next bottleneck, or am I wrong here. As I don't have a great understanding of game engines and server tech, I don't really know if like the entity graph update is 100 times less compute intense compared to the engine work (physics etc). Hope this isn't a stupid question...

[@free-dissociation]

You're right, this *is* gonna be a limitation, and it's gonna depend a lot on specific implementation details. I unfortunately don't feel like i have a good intuitive sense of where and when it's likely to bottleneck. Updating the entity graph on a transaction-by-transaction basis is definitely less compute intensive than the physics simulation, but because the global entity graph has to be able to receive and process updates from *everybody* currently playing, not just the players in one region of one shard, it's still a hard problem. Really the best practice with optimization stuff like this in computing is just "try it and find out."

[@sc_cintara]

@@free-dissociation I think the answer to this is that the entity graph is not the replication layer. The replication layer might be caching things for the entity graph. The replication layer is also redundant and horizontally scalable, and the entity graph is the persistent storage. As long as the replication layer is able to scale horizontally, it can take the load for tens of thousands of simultaneuous players in a single shard.

[@Crispy_Steak]

the replication layer uses a queue to update entity graph I think, per 2021 cit con presentation at least, shoudn't have a fixed (30 ticks per second) update rate that way. Your other question I don't know enough about how this works to answer.

[@EdsEnemy]

i don't have time to listen to this right now but chapters or timestamps in the description for the main points you wish to make would help a lot

[@adamchambers1393]

They have had bugs on every demo they have done for their live stuff tbh. Their video stuff course not so much. But even the pre-recorded video for Star Engine was buggy which why they been fixing it since :)

[@sodiufas]

Oh ok, another thing about SC is projectiles physicalized, that's why we doesn't have any hit scan weapons.

[@TheNerd]

The prinicples oft this technology could change how we think about multiplayer.

[@realmwatters2977]

We are star citizen players and we want your opinion on server meshing. 07 see you in verse!

[@Austintatious_One]

I think alot of changes of aproch over the years have come from a shaky architecture.. Originally I think the engine and data sets where being optimised for optane, possibly with SPDK.. Now though I think they have had to migrate to CXL, this will possibly be better latency and scale wise, definitely it will be better for support and implementation going forward though.. 👌 Also.. Shooter calls the shot I don't think will be relevant as the shots are raycast and outcome it self is authoritive and know instantly by both parties.. 🤔 (Will still be a micro second faster for the party taking the shot but not in any relevant way compared to older z axis rounding calculations which does have its inherent issues in group play)

[@steved4475]

They need cross multiple server interactions to work. Shoot from purple past green and into red. That would be the case of Shooting out the MSR cargo bay through space and into the Idris hanger to hit a target.

[@mobiuscoreindustries]

It's likely that server boundaries will greatly extend past any individual interaction so that in that particular described case all entities would always be on the same server. To the server however at a macro scale all information is the same. So it may not need to account for someone shooting across the boundary but it may encounter hundreds of ships, their cargo and items crossing the boundary, all the while the players inside are actively doing things. Not counting NPCs doing the exact same.

[@sc_cintara]

@@mobiuscoreindustries You are right, but only in the first step, static server meshing. Once they get dynamic server meshing (which they have to do since they hope to have a single shard per geographic region), they have to split up the tens of thousands of players per geographic region into their own 100-person (or whatever) servers. That means that the server boundaries will be really tight. They need to expand to many more landing zones too, or it will be unbearable to go shopping! 😂

[@sketchtherapy1218]

Would you please do a video with Mr Tybio about the 5 toughest problems in computer science CIG is taking on or 1 video for each of the 5 with Mr Tybio?

[@AlbertoMartinez765]

Its Nice to hear from someone who is actually Knowledge about the subject.

[@DaeDroug]

My impression was that the red, green, and purple servers where running on 3 different laptops for the demo. Which having it working like that rather than over whatever in rack inter server bus they might be able to utilize is especially impressive.

[@madsmith1352]

No, the system (whether a laptop or a more powerful full pc) is spawning three server processes running locally on the system. There’s no significant network latency being incorporated in this demo as none of the network communication packets need go out over the wire.

[@sodiufas]

@@madsmith1352 it shouldn't travel far in cloud too, why would it have to?

[@jackinthebox301]

As Madsmith said, in this demo you can see him click in his dev program and change how many servers are being created. The whole demo is being run off of one PC. It's also entirely reasonable to expect that their plan takes into account where the simulation servers are located in relation to the replication layer servers and the latency that will inevitably introduce. These are network engineers after all.

[@sodiufas]

​@@jackinthebox301 they are basically should be in the same datacenter. Why there will be latency, thats my point. U can go on their website and look where are they located on the map.

[@sodiufas]

it was not, they had servers at stage, but no way laptop could run it@@madsmith1352

[@x0myspace0x]

I'm not really surprised that it works in a test setup. You can get anything to work in a test setup. What I'm wondering is whether it will keep working when this gets upscaled to thousands of simultaneous players and millions of individual objects being streamed between different clusters and the interactions between all those entities. Time will tell...

[@mr.potato9449]

As someone who doesn't know much about programming, what does this mean in regards to server player limits? Say the server limit is 100 and call this demo a city. Without meshing you could only have 100 people in this city so would only see and interact with 99 people. From what I gather of it, with the city now being 3 zones and a different server having authority over each zone then you could have 90 people in the purple zone and 60 in the green zone so if you were in the purple you would now be able to see and interact with 149 people and not just the 99 you could when it was just 1 server with no meshing. Is that correct or am I way off? If that is correct then the servers still have the 100 limit so what happens if there are 100 people in the green zone and you move from the purple zone to the green zone?

[@free-dissociation]

You're correct! The goal here is to be able to substantially increase the player count on what we've called "servers" up to this point-what CIG are now calling it a "shard", and it is made up of multiple servers. Ideally in the case you describe you can see and interact with all 200 people (100 in the green zone and 100 in the purple zone, each on their own server) seamlessly as though they were all on a single server.

[@mr.potato9449]

@@free-dissociation So what will happen when you move into a zone which is already full? Is there some kind of dynamic system where it will split that full zone into smaller zones (which now have their own server and limit) until there server in control of the area your entering is no longer full? If something like that is the case I would think there would have to be some kind of noticeable latency with the amount of times they might have to move people from server to server if an area is full and after the split other people have already entered it making it full again and needing additional splitting and so on o.O

[@carllong8954]

"distributed system is the hard part" That's why we have all been waiting for this game for 12 years now. I can remember way back at the beginning when CR was talking about having 2 cap ships fighting and each was a server and shooting from one ship to the other would involved hand offs between 3 different servers at the shot moves. That is why this demo might be the biggest step in the development of this game. This demo showed everyone who understood and has been waiting that it may actually be possible.

[@a__random__person]

in future, since you're recording a youtube video in 4k.. you should either upload in 4k as well so we can read the text in the video easier.. otherwise reduce the source video to 1080p :)

[@St4rQu3st]

You could pool objects such as bullets so they are just reusing bullets instead of creating and replicating them across. Just pool and use them and send them back to the pool.

[@ThorbjrnPrytz]

They run on Amazon cloud servers, And the internal high speed network on the datacenter should minimize latency between servers

[@thepurpleperp4564]

I would expect that the majority of the boundaries between servers will be out in empty space, not in the middle of an area where high action fps pvp is generally happening, no? Not only does it reduce the number of cross-overs at any given time but there will rarely be anything complicated or important happening at these boundaries. Probably only quantum travel. EDIT: Also, isn't it possible that there's never a situation where you could even shoot across 2 servers?

[@free-dissociation]

It *is* possible to set this up such that there's never a situation where you could shoot across two servers, so if they're demo'ing that ability then they must expect it to matter.

[@BroodlingX2]

There ultimate plan is to have large ships with 50 + crew be its own server. Basically a server inside of a server. So projectiles would need to be able to cross servers since every time you shoot at a large ship it will need to go into another server.

[@thepurpleperp4564]

@@BroodlingX2 Roger that, I was talking more about shooting from one server, through another server, and into a third. Addressing the L-shaped demo.

[@josephbaker6461]

The bullets in Star Citizen are indeed individually modeled, probably for the reasons he said. If you play with the active camera in game you can see it.

[@crispy9175]

Is the reason they are using a graph system to store data like this due to the extreme amount of entities and objects that they are trying to track and keep persistent? And why can't an entity on the purple server shoot an object on the red server (excluding the corner they'd have to shoot around)?

[@madsmith1352]

It’s not a question of “can”. There’s a lot you “can” do. It’s a question of how complicated the problem becomes and what errors can manifest from that additional complexity. When you’re shooting from server A, through server B, to an object on server C. There’s three parties that now have a vote in the success of that action. Server A determines if the shot happened and what angle and velocity the ray or projectile is fired. Ultimately server C has authority over the target of at the time of hit if the ray or projectile intersects with the target and computers what damage should have occurred. But because the shot goes through B, it may have to compute intersections with any of its authoritative entities and the path of the ray or projectile on each tick of that shots motion. The handoff is from A to B to C. So there’s another set of handshakes of authority handoff that could occur for each shot. And depending on the latency of those control handoffs, creates a larger time window where unexpected behaviors could manifest due to desynchronized views of world state between the three servers.

[@crispy9175]

@@madsmith1352 I don't think that's correct. I've seen a few devs talk about how many systems use the client that shoots to determine if there is a hit. Overwatch 1 didn't do this and had problems, overwatch 2 solved the issues by having the client "call the ball" on whether a hit was achieved. The same could be done here with the client server/shard calling the hit on the target.

[@EwanMarshall]

the purple server was suspended and no data was going to/from it from the others in the demo. Basically it automatically suspended itself to save server resources. The idea is to only track what you need and not the entire physics region. It is similar to how my comment here may well be being sent to an entirely different server in a different worldwide region to you. But there are probably google servers that never even seen this comment data because no user hitting that server has been on this video. The corner was deliberate to induce this condition.

[@matthewvelazquez2013]

So, uh, At minute 35 the inference I'm making is That the client can crash his game By spamming The activation and deactivation of possible In-game surveillance drones. Maybe the simple solution is to insert a card Displaying the words acquiring signal In the mobi glass of the player character - Giving the game engine actual time to call on everything contained in non adjacent locations? Would not the spawning of a possible surveillance drone Initiate the same rules used for a player character? Could a nefarious user then rapidly spawn The multiplication of in-game surveillance drones To crash the server?

[@rakaydosdraj8405]

A drone with a camera doesnt need to see anything unless it's being operated, which means you can stow the operator's perspective and jump to the drone's "remove view" perspective, which takes up the entire FOV. You dont need to have both in operation at once, which limits the number of observers in the system to the number of players.

[@matthewvelazquez2013]

@@rakaydosdraj8405 thank you for the specific insight. But, this IS Star Citizen so maybe Chris has 2 Savant developers working together in a 'Hidey Hole' somewhere to bring 'picture-in-picture' surveillance drone operation into the mobi glass interface? Heh-heh-heh. Not implausible, considering what they've been able to do so far...Ha ha ha ha...

[@rakaydosdraj8405]

@@matthewvelazquez2013 It IS star citizen, so Remote Turrets already exist as a thing, in live patch. the "full field of view" workaround seems to be what they're going with. Generalizing that system to recon drones is a cleaner solution.

[@matthewvelazquez2013]

@@rakaydosdraj8405 thanks.

[@doubledigital_]

player can see it because the replication layer is now showing all clients all the logic ,, but the server its self has streamed out the data.. saving a fuck load of memory the replication layer

[@sodiufas]

Why there will be huge latency if both servers and replication layer using same infrastructure inside same claster?

[@free-dissociation]

"Huge" here is relative to the latency of doing this in the memory of a single process. The rough rule of thumb in systems is CPU registers -> CPU cache -> RAM -> Disk -> Network, each step is an order of magnitude slower. Lots of asterisks as tech has evolved but that's the intuition.

[@Quent1nB]

My thoughts and question: - bullets: while they are going through the RL so that other servers see it, I doubt they are going to the PES graph db (= no write spam), I assume they ban some entities that have a short ttl and no purpose. - your comment about latency on real servers is true, however a DGS will usually serve maybe a system, a planet, or a city. Cases where it will be assigned to a space ship will probably be rare, and I doubt they will go "lower" than that, so we will rarely observe any latency from that. If it comes to indeed splitting ships/UGFs into multiple servers, I'm sure there are some solution (your idea with k8s pods on the same machine is a good one tbh) - question: when sending objects between two adjacent DGS, I understand that both have a copy of the object, and only one has authority. Do you think both DGS run the simulation for that object though, when they are close to the "border" at least, so that handing over the authority is as smooth as possible? Thank you for your videos!

[@free-dissociation]

> - bullets: while they are going through the RL so that other servers see it, I doubt they are going to the PES graph db (= no write spam), I assume they ban some entities that have a short ttl and no purpose. Yeah agreed that bullets aren't going to go back to the global entity database. Even there though the write spam on the replication layer strikes me as high, and my read is that they *have* to go through the replication layer. It's possible CIG have worked out some hack with raycasting as other commenters here have suggested. > - question: when sending objects between two adjacent DGS, I understand that both have a copy of the object, and only one has authority. Do you think both DGS run the simulation for that object though, when they are close to the "border" at least, so that handing over the authority is as smooth as possible? I think naïvely I would try without this, just doing the handoff between physics simulations when the authority gets handed off, to simplify the process. I can imagine there being enough randomness in certain kinds of physics simulations that two servers might simulate the same object very differently, but there I think the better solution is to make the physics simulation behave as close to identically across all servers by e.g. stuffing the necessary random seed into the object data.

[@Quent1nB]

@@free-dissociation Oh good point about the randomness of physics sim. I guess we'll never know 100% how things work unless someone from cig tells us, but I would love to have all these details. It sure is fascinating to watch your channel and see how someone else thinks. I work in IT but nothing to do with gaming.

[@craigwoodward8455]

7:16 CIG dont have a habit of avoiding bugs during live demos... where'd you get that? It's on the bingo for one of their live demos to crash. XD It almost always messes up for them, the amount of times its crashed entirely during the demo has been a meme. lol

[@absentminded02]

The statement i heard from people at the event was Chris called the team cowards if they didnt do this live. The fact that it worked at all is really promising

[@sodiufas]

bullshit

[@absentminded02]

@48:30 I want to point out that, He's not running all three servers on a single laptop, There are 3 laptops back stage that are running the Servers. This was clarified by people who were at the Event talking with the engineers after the panel.

[@sodiufas]

​@@absentminded02 yep i agree pretty much

[@Crispy_Steak]

one of the servers locally was on that machine for certain, it bound to localhost in the logs when starting up, failing to bind to at least 2 ports (12300 and 12301) at least when the 1st instance booted. I'm pretty dang sure this on the machine on stage, the 3 dgs mesh instances use just under a 5 GB memory each at the start with the minimal environment being loaded. Technically they probably could have, but having a bunch of weird machines running a demo wouldn't be something I would be comfortable doing on stage.

[@czbombah]

Security IT engineer talks about game :D

[@sevilnatas]

SeconfLife has been doing this for a very long time. Not saying that they do it better or that their engine would be up to running such a complex game like Star Citizen, but there is something to be said that they had this distributed model running more than a decade ago. I wonder how the two solutions compare?

[@CSKefka]

They do not.

[@sevilnatas]

@@CSKefka They do to.

[@free-dissociation]

Yeah, EVE Online as well. I know a tiny bit about EVE's internals and nothing about Second Life's and would love to talk with people who know them better. EVE at least does handoff between servers at the solar system level, and it's pretty clear to me that CIG want and need something much more fine-grained here.

[@sevilnatas]

@@free-dissociation Yes, definitely finer grain for CIG. From wikipedia "Each full region (an area of 256×256 meters) in the Second Life "grid" runs on a single dedicated core of a multi-core server." I was originally thinking they were doing a region per server instance, but it looks like it is a region per core. Interesting. So the big explosion of core per cpu, probably really benefitted them. I have a dual cpu EPYC server running in my rack at home that has 48 cores total, that I pieced together off of eBay from Chinese decommissioned hardware for like $1k. It is like 3 generations old EPYCs and used RAM, but it runs like a champ and I guess it could run 48 regions of SecondLife, crazy!. I imagine their server footprint, to support the same number of users, gets smaller every year.

[@free-dissociation]

​@@sevilnatas And in this day and age individual cores have gotten so performant that it's entirely possible they could run multiple server processes per core. That was one of my surprises from the interview I did with Zach Johnson-he said that Diablo II Resurrected was able to run that way: kzbin.info/www/bejne/i6bTgal-pMiIh8k

[@TB-4]

Just figure out how to design it after The Flower of Life and it will be completely balanced

[@Yokeus]

I would be curious with how big CIG decides to make these zones. If the red zone is 1 Million meters across for a whole planet, you wouldn't be able to shoot across three zones like you had mentioned. The weapons and scanning don't have those kind of ranges at the moment.

[@doubledigital_]

the solar system its self is a giant zone, they can nestle everything .. they just need to link them all to servers now.. fiddly but its more or less done :))

[@80KBM]

I´m really curious on the full implement on HDR, DLSS 2.0 (3.5) Raytracing and Vulcan. Did i remeber correctly that they said q4 2023 for those thingis?

[@surslaughter3524]

Great video. I have one issue though. Saying CR didn’t risk anything when speaking about anything in SC is wrong. He put everything on the line. From most of the gaming community screaming scam for 10 years. Half of the SC community doing the same. Every issue, every marketing disaster. Every failed patch. Every projected release date for ships/features/patches. He put his name and career on the line every day for 10 years. He deserves some credit.

[@miinyoo]

You can see the transition latency at just before 56 minutes in when Ben jumps crossing into the red area. That is going to be a real trick to hash out further complicating the already complicated. Doing that a thousand times per tick in say a fleet battle that moves from one zone to another is going to be monstrously difficult without wasting a lot of resources. A better idea would be to dynamically adapt zones (In a give and take between servers) based on context and call load something like you mentioned on a protocol level. In space, you can see as far as your angular resolution will allow. With fast projectiles you can touch them. In order for large fleet battles to be possible, I can't imagine any way to do it but with intelligent adaptive zoning. Easy peasy lemon squeezy for a single player game. Whole different animal for MMO. Perhaps AI could be of use as a co-processor to the replication layer to adjust zone load dynamically and feed that back to the actual servers and clients (without hallucinating) while doing that in literal milliseconds. Latency budget of any game is as close to zero as possible but if you get to 100ms territory, you've already broken it. It has to feel like at least 30ms latency or lower on client assuming you have a perfect connection. The idea is super cool and that it is pretty much seamless is not trivial, I imagine, but holy hell what bandwidth requirements that must balloon into when you're trying to simulate this granularly. I feel like doing all of that and keeping it seamless is like borderline insanity pushing the limits of the actual internet's infrastructure especially in terms of latency and stuttering problems like Ben's border crossing. That doesn't even include gameplay "what actually happens in game" problems they have to solve. Solid take. Kind of confirmed what I already suspected or at least we have the same suspicions, yours obviously more knowledgeable. I love that there's an Assassin's Guild at MIT of LARPing nerds who pull out the stops and make everything feel super immersive. Makes me think of optical circuits and their potential. In phase 1, out of phase 0 not just one at a time but for each band of wavelength. The transition is the trigger just like a very very tiny interferometer. How to apply those principles to something as stable as silicon. Hmm... Maybe some day we'll have x-ray or higher frequency circuits and look back on the archaic use of the electron's influence to do calculations.

[@Think666_]

For those who don't speak Engineer "brave choice" = "bad choice" unless you know something they don't...

[@doubledigital_]

the server is running on the laptop that is true but the fact they even have it working will unstress the servers .. we can have as many servers as we want now.. not just one getting hammered lol nu us lot.. you can even apply this to space battles if needed so when a fuck load of people are in a small space you just spool up enough servers to deal with the traffic bobs ya uncle ^_^

[@kemo_l7350]

How could I get a job with cloud imperium games?

[@kemo_l7350]

I mean that I don't have that many skills and I am in Jordan so the only place I can can learn from is the internet

[@oskar1799]

Hey man, love the idea. I have some note´s, Keep it short prepare a script, use jump cut´s and dont stop to often about some small detail´s, try and translate the thing being shown into "Human language" so the normal person understands what is being shown and if it really impressive or just marketing. Also maybe expand from only SC and go into other game´s and it´s game engine feature. Keep it up

[@sc_cintara]

These are good suggestions, however I am not sure about the translation into "human language" bit. There are many youtubers that try to present these technical things in ways that everyone understands, there isn't really a need for another. This video is special because it doesn't do that, it digs deep into the details, stays fully technical and speaks to developers and engineers. That is valuable! One should always be as simple as possible, but never at the cost of fidelity. If you are afraid of mentioning something like PAXOS/RAFT, you can't get the full message across. So I think there is good value in keeping it fully technical, we have enough of those that speak to everyone. I would like to see a youtuber take the engineering part of the monthly progress report and talk about it in a technical way.

[@Whoisitthatknowsyourthinking]

we don't want them explained in basic terms dude. Most people watching this video want a high level explanation of what may or may not be happening between different services etc.. It's refreshing for me seeing as my line of work is Devops/AWS solutions architecture and I deal with things like (EC2, ALB's, Lambdas, RDS, mong DB , EKS, the list goes on ). I want to know at a high level how this is all implemented in the gamming side of things which I don't have a lot of experience with. I do get what you're saying but there are 100 videos that are layman's terms that basically sum everything up in a minute and a half with answers like "its the future ! you can have as many servers as you want ! thousands of people in one server ! It will change gaming forever! Capitol ship can be a server ! Magical tech that only CIG can do ! Ect , ect ... It's extremely important that we have people in the community with a very high level understanding of the kind of Network architecture / software it takes to do this, that way people that are less techniacally inclinded can manage there expectations a bit in terms of what all this tech can and can NOT do.

[@j.d.4697]

Opinions from people in the industry on SC are rare but we sure would like to know what people think other than CIG themselves or all those people who are 100% opinion and 0% knowledge.

[@BradAhrens]

Subbed because I wanna see you lose your damn mind when it's in-game! 😂 GO, CIG!