A Software Engineer Reacts To Star Citizen Server Meshing & Replication Layer - CitizenCon 2953

[@CSKefka]

Apparently when the guy who made this first showed Chris Roberts, he was told that if we wasn't a coward he would showcase it live. Well done.

[@tenthlegionstudios1343]

Game dev system design is nuts. Low latency multiplayer games where you are trying to sync everything in real time is a hard problem. Thanks for the content!

[@RealKanashii]

IT crew use to say: there's no real time, conceptually and technically. So yes, real time is always a problem (and that's why they use the "ticks"). :(

[@edward3190]

it's impossible. SC will fail no doubts. The reason MMO games use regional servers is that the speed of light is finite. Light travel from Asia to America takes more than 100ms, which is enough to destroy competitive gameplay. SC choose to ignore physics is beyond insanity.

[@claiyc]

@@edward3190 misinformation my friend. The goal of sm is not to overcome latency problems in terms of "letting people that are on other sides of the world play together", but rather to just distribute load by splitting simulation of a gameworld into different processes that can run on different machines. Everyone is well aware that latency will always be a problem, that's why it's highly expected (by devs aswell as community) that there will always be regions (NA, EU, ASIA, ...) in SC. Having ONE global shard is an existing idea, but no one guarantees that smth like that will ever exist (perhaps there are options to achieve something with instancing and concepts of "consistency over time" but that's it from my pov - no real time gameplay interactions between two clients that are on other sides of the world)

[@nicodemiusoverschnaps4759]

distance is approx. 10.000 kms at 299 792 km/s... more than 100 ms.... Suuuuure. ROFL.

[@edward3190]

@@nicodemiusoverschnaps4759 just shut it if you don't understand real life. light doesn't travel that fast on Earth, and 10000km... wtf?

[@sc_cintara]

The distributed lock and all that is cool, but the really big job was that all state had to be externalised from the game code and copied into the replication layer. That is *all state*, because a server has to be able to take over in-flight or take over from a server that crashed at any moment. So essentially, server meshing changes the whole code base, all code in the game servers. I saw another hint in some other presentation about how things work: They said explicitly that the entity graph is distinct from the replication layer. I agree that the entity graph might well be Neo4J, but I think the replication layer is something else, probably in-memory. Also, in last year's Chairman letter, CR mentioned that the replication layer is horizontally scalable and redundant, so the replication layer does need to handle distributed lock and replication in a cluster. BTW, I have founded dev companies so I can guess why CR cried when it worked: CR sent off this team to build server meshing many years ago knowing full well what the job would require (he is a good dev on his own merits; he wrote most of Wing Commander himself). Starting a project the size of server meshing is a "bet the farm" project. If the dev team had not been able to get server meshing working, CIG would quite likely be over. So CR had to trust that the dev team were able to do it, entrusting them with the future of CIG and let them forge on ahead through all the delays and the failure of iCache. That is a nerve wracking position to be in when you are responsible for 1200 people's livelihood!

[@JohnMcclaned]

could be memgraph

[@OrniasDMF]

and when people are constantly calling your game a scam

[@coolkid15105ify]

Thanks for your insight and context :)

[@gordonhowett7529]

Dude this was exactly the kind of professional opinion I was hoping to get from this video. Great work, great insight. You got another subscriber here!

[@zf4120]

"It should not be possible" is usually followed by "How did they do that?"

[@eltreum1]

Yup, like SpaceX and self-landing reusable rockets everyone said would never work.

[@domisPL_01]

​@@eltreum1 Do they work? How much cargo and how many people did they transported?

[@eltreum1]

@@domisPL_01 They work very well with multiple crew and cargo missions to the ISS and dozens of satellite launches. One booster was re-used 23 times before it was no longer usable and about 50 reused boosters in the fleet total. Thats 50 primary stage rockets not dumped into the ocean after 1 use. They have already reduced LEO launch costs by a factor of 10 with 3 companies now scrambling to copy their success.

[@domisPL_01]

@@eltreum1 ok, thanks for info 👍

[@Hax187]

50:46, the bullets are ray casts which is incredible to me. There was a interview with one of the developers and he said that raycasts are working but projectiles are not working atm

[@Crispy_Steak]

sorry, do you happen to know anything more about this interview? Whose interview or a link?

[@thenecrolept]

Don’t have much to say beyond, very interesting commentary, very cool insights. Definitely subbing in case you do more similar content in the future. Or if you get that interview!

[@artlife9563]

Well this example is a very small room. The key is that currently, one server has 5 planets and several moons. Imagine if one server only had to account for one planet. The data strain will be greatly reduced. The transfer latency between planets or moons wont have to be instant as shown here, because planets are so far apart.

[@jafogx]

Yeah this is something that I wish they had made more clear. The way I’m picturing this is in the game there is no reason to have three different servers handle three different rooms in the same building, this was just a scale model to show the technology. Ideally, a server would take care of a large area (as you say, could be a whole planet, a station, etc) and then we wouldn’t notice any lag because we’re in quantum when it happens. That would also mean the example he mentions where you’re shooting something across three servers is infeasible because of scale. They don’t mention this specifically on the conference though, so it’s kinda up to our own interpretation. They could’ve said “these rooms would represent different planets, or a 1/4 section of a planet” or something to give a sense of scale of how it would actually look in the final game.

[@uttrik]

Though the end goal is to have the mesh dynamically scale based on the number of players in a region. If a thousand players show up on microtech, multiple servers are going to have to take care of a single planet. And if they all show up at new babbage, there may very well be situations where rooms are divided up into separate servers. Maybe the designers will make it so server boundaries will never have situations where you can shoot through multiple servers when indoors.

[@free-dissociation]

Because they demo shooting across server boundaries I kind of infer that they actually intend for that to be a thing in the medium term.

[@_R8ze_]

​@uttrik this person gets it!

[@grygaming5519]

@@uttrik Oddly this is something MMO's have to deal with and are unable to deal with. Case in point just look at Final Fantasy XIV Endwalker's Login issues, 5000 players all trying to log in at the same time the servers were unable to handle it. Now if this teach was in its growth/maturity stage something like having login issues for MMO's or dealing with crowded areas would just be a thing of the past because the system would handle the issues without a problem.

[@originalgameronline3457]

My man had a Stargate reference in his video. That always deserves a like. o7

[@dmacpher]

And Enders Game with the Ainsble

[@free-dissociation]

Originally the ansible was Le Guin, from whom Card borrowed it, but yes I am a big ol' nerd and have been forever ;)

[@dmacpher]

@@free-dissociation Ursula! Didn’t know lol

[@Blu3Souls]

I do whole-heartedly believe that the dog-leg is there because they wanted to show the stream-out mechanic. If they let us shoot from one region into a different region across a third region then the two outer regions will probably not stream each other out. How that works with dynamic server meshing where you can't know what zone has "view" into other zones will have to be decided (I'd bet on distance between the zone edges)

[@dawnphantom2439]

I think the idea is that at scale, one server is Microtech, the other server is Arccorp, and the third server is Crusader. The distances between server zones is so vast, that this won't even matter to begin with and the idea is you'd only ever need to interact with things in the nearest server over anyways, and never need to reach through the next server to reach the third.

[@bambooza1]

I agree the dog legs purpose was to allow them to showcase what happens when game objects go out of view. One thing I think keeps getting overlooked is that from the engines point of view all movable objects have an origin point in the parents reference view. So when you move an object you only need to update the objects origin point, and not every point that makes up the mesh. The boundary box for each server mesh then can be a simple coordinate check if that point has crossed the boundary and can be handed off ownership. This prevents a lot of calculations and the issue of portal slices of game objects. Then the players graphics card can do the heavy lifting of matrix calculations on the objects mesh to camera view. It is also more than likely the same values used to reference in the replication layer and storage. You just need to know the object ID, the object origin point in parent coord values, parent object a list of child objects, and list of mod values. I know they said it's being stored in a graph database but given objects can't be inside themselves there is no potential for circular paths and so it's just a tree problem.

[@sodiufas]

this show is only meant to early wh, oh baker reassurence

[@ThatOneCakeGuy]

@@dawnphantom2439 The end goal is much smaller, but in the demo they kept it limited to the 2-3 fields which is just a place holder. To start out it's going to be one system per node, then likely planet, then probably regions of planets, POIs, etc. down and down as the tech evolves and population increases. The end goal is for it to have "as many as needed" to handle the population of an area. Ie if one area is more popular it may have several smaller zones with each master having a further view where as a low pop area might only have one zone covering a larger area because less people go there.

[@gruller65offroad96]

Remember thease zones will be significantly bigger , think station , quarter solar system size to this issue isn’t really a big deal , only pes is the biggest thing

[@Billy-bc8pk]

At 45:40... they're handling it via Entity Zoning through their SSOCS infrastructure, which is really impressive tech. Interestingly enough they were testing the server mesh via Entity Zoning back in June with limited Evocati participation just to ensure that it actually worked. The instructions were to have 200 players move from one specified object container to the next (I believe they started at Everus Harbor and then had to quantum to another station/planet designated as a separate zone) to see how well seamless transfer authority worked.

[@juz3r1]

Such a test is currently underway at EVO!

[@artuno1207]

"This is not something a AAA publisher would have put up with" this is why Ive believed in this project. This why ive been patient. Theyre doing soemthing phenomenal. I want them to succeed so that all the haters of this project and those who call it a scam have an amazing game to play soon(TM).

[@sc_cintara]

Yes, the funding model is the real secret sauce of Star Citizen. No financer would approve of waiting this long to start selling the game if it had been done in the traditional way. All that money sitting in CIG would be money not giving of interest, so the expected profit requirement for the game would grow year on year until they become untenable. At that point the game would have been forced to publish to recoup as much of the loss as possible. It is the main reason why Forbes hates Star Citizen; the funding model entirely cuts out the financer.

[@r2artCH]

I think the game already good with all the current features. Only one system but there's so many things to do. There is only two things that bother me with the current state, and that is the bugs and the NPC AI. If they fix those two, I can say the game already complete even with only one system. With this player already able to play the game, while they working on expanding to a new system and new vehicle internally until it's ready, stable and minimal bugs they can update it then continue expanding internally until ready. Just keep doing it like that, it should be good.

[@sc_cintara]

@@r2artCH Unfortunately, having each release mostly bug-free will slow down development quite dramatically. Think of it as building a something like a chest of drawers. You build it with some group of drawers to begin with, polish it, make sure all the nails are tightly nailed in, etc, and give it to the customer. Then you want to add new drawers. To do that you have to break apart much of what you already built to fit the new drawers. Then you have to re-polish it, fix it up, etc to make it look nice again. This is the problem with releasing code under development: If you want it to be bug free and polished, you have to redo the bug-fixing and polish every time you release and that slows down development dramatically. So CIG are doing minimal polish and accepting quite a lot of bugs, especially in systems that are planned to be replaced, to avoid getting all resources stuck in bug-fixing and polish.

[@AlbandAquino]

50:45 To answer your question: Yes, they are. But instead of having a 3D mesh to represent the object it's treated more like a physicalized particule (GPU bound). So you won't see a mesh around those. But they are physicalized. 59:19 That sound like something you went through 🤣 I'm a DevOps engineer, specialized in IAC. I felt that moment... 😅

[@toecutter9372]

what do the blue and yellow colors mean for your pic?

[@AlbandAquino]

A sea of sunflowers under a blue sky. But realistically, the Ukrainian flag. @@toecutter9372

[@MaticTheProto]

@@toecutter9372something good

[@free-dissociation]

> That sound like something you went through 🤣 I have stories. So many stories ;)

[@ITHYANDEL]

I think it is not using a physicalized bullet, but a time constrained spline, the spline would then be subdivided for each server through the replication layer, which keep the full spline.

[@bambooza1]

Ya they said they went back to hit scanning from physicalized bullets.

[@budthecyborg4575]

Very insightful about the handoff lag. The best way I can think of hiding handoff lag right now would be to make sure server areas are all sized well beyond the practically implications of player input, relative to the size of the game the range of your bullets is nothing. Forget about three servers in one hallway, the smallest server zone in v1.0 of server meshing will probably be an entire planet/moon, areas separated by millions of Km. I would expect handoffs to only take place in the travel between planets at which point the likelihood of players engaging in combat on the border between servers would be effectively zero. As-is we're limited to about 40 people in each instance of the entire solar system, giving one server to each separated POI in the system would allow an absolute maximum of about 1,200 people (there are around 30 individual points of interest in the Stanton system). Of course the total would still be limited by the single highest traffic point in the game, but I would expect at least an order of magnitude more players active in the system at once, once server meshing is in Star Citizen will finally feel like a real MMO.

[@iiKiboshii-c3z]

Current instance pop is a flex number of 100. They have had 200 player servers.

[@budthecyborg4575]

@@iiKiboshii-c3z Excellent, my only info was from a gameplay video and I assumed he was correct. If CIG servers are good for 100 players per server then Server Meshing is going to make Star Citizen really pop off.

[@mobiuscoreindustries]

​@@budthecyborg4575 the thing is we know somewhat they they can already as people have stress tested the servers by bringing all 200 people of a server into a single ship. And barring restricting people to walk (so people would not die off running into each other and colliding) the servers were actually running better than normal in this scenario. It's because right now the cause of server load is due to all the players being spread out everywhere, meaning the server has to load and account for basically every major landing zone and most points of interests at the same time. All those interactions stacking on each other until something bad happens. Essentially, CIG's streaming tech can't be utilized to its fullest server side as long as servers functionally will have to keep most areas loaded.

[@free-dissociation]

Yeah, it's certainly possible to set up server meshing so that it's very rare for players to cross servers and thus hide server leg. Because they demo'd interaction across servers, I infer that they actually intend for that to work and be a big part of how the game works at least in the medium term.

[@budthecyborg4575]

@@free-dissociation My bet is they need localized server meshing to work before we can have a playable Bengal. Players have speculated for years about a single Idris meshed across multiple servers but that would never actually be necessary just for an Idris, the Bengal on the other hand is likely to rely on this system.

[@MrJabbaStyle]

this is the stuff that i don't understand at all but still understand enough to be highly impressed!

[@honordevs]

What a lot of laymens don't understand is that the entity view is actually just the graph database. It's an advantage of using one, like neo4j, that you can easily present the information. Most of my friends thought it was a custom tool, (which is definitely is modified, but i digress) that they made for debugging because they had never used a graphdb.

[@RedHornSSS]

Very unique take on the topic. Good to hear from ya

[@sharxbyte]

this is an incredibly intriguing as an engineer and amateur programmer! Also thanks for introducing me to the term "Nerd Sniping". Love Munroe

[@ParaquatSC]

A note that each game client around @50:00 shows how many entities are tracking and the act of shooting doesn't appear to affect that counter at all. So bullets would appear to not be entities.

[@vorpalrobot]

The projectile system is different than the one that handles objects. I think they might have tried to physicalize bullets etc early on, but I remember them excitedly showing off the new projectile manager with a big fleet battle with thousands of shots in the air.

[@ParaquatSC]

@@vorpalrobot definitely. Also tracking projectiles moving at 1000mps+ sounds very complex once you add latency!

[@Dovahkiin53]

Thanks, well explained and thoughtful.

[@richardmoore4194]

Is managing these transactions in a node DB compared with RDBMS so much harder , or is it that we have 40 years experience and mature methods and patterns associated with managing RDMS. I am so old I remember switching from PICK to Dbase III early ORACLE had all sorts of undocumented features. Thanks for doing this BTW , i much prefer this level of insight with house mates making late night snacks than glossy videos that gloss over the issues 🎉

[@eltreum1]

Speaking from experience of converting an RDMS to GDMS It's a little of both. SQL has the advantage in write speed for realtime transactions especially if you don't have varchars fields and fully rigid rows of reasonable size. A true GDB like Neo4j destroys SQL in search/read speed regardless of database size the moment in SQL you would have to use the Join or Union keyword. The way you model and query the data are very different mindsets between the two when designing them and your apps.

[@richardmoore4194]

@@eltreum1 So RDMS are faster for ACID transactions, but GDMS are better at reading records and significantly equally fast irrespective of size of the DB. So will some sort of caching / queuing system be needed. Providing it was a real working demo - the focus will be to make it stable - then make it fast.

[@eltreum1]

@@richardmoore4194 A couple years ago they implemented what they call icache as a caching layer that is the foundation to the replication layer. It allowed servers to crash and us not lose ship and cargo every time if our cargo made it to cache before the server crashed. GDB isn't drastically slower on writes, but you do have to plan your workloads if you want consistent speed per transaction. EvE online did a similar thing to address the DB load/response time. The entire DB was replicated in a mirrored pair of RAMSAN cache arrays connected to the server cluster via Fiberchannel backplane. The RAMSAN also had read replicas on SAS storage arrays for backups and read queries that did not need real time priority. NVME and new gen RAM storage arrays now eliminate disk seek time mostly if you need it. SC is a little different and a game that will play itself. There is a universe simulator that is just a machine learning system that calculates the macro economy and state of the universe stats and designated target metrics and employs NPCs to balance things as players affect systems. That is mirrored to/from the replication layer and other backend services that are like APIs in a microservice system with the replication layer as a data broker. The game servers will be assigned authority over parts of the universe and any players or NPC that travel there so the live state is ran with distributive computing. NPCs the players can't see are running as microservice agents on other server pools streamed off the gameplay servers interacting with the universe system at the speed of a player for travel/work through APIs to interact with the replication layer. When NPCs are in sight of a player they are streamed back onto the gameplay servers so you can see and interact with them. That is part of the plan to get capacity and scale is offload the gameplay server anything a player is not going to see so the universe keeps moving. Crimes and cargo NPCs keep working to modulate the security and economy to a target in response to players messing with things. Sorry for long post you should watch this vid where their engineer shows the stuff in action. kzbin.info/www/bejne/aJ7YeIqqjZ2Ybck

[@eltreum1]

@@richardmoore4194 They are essentially using something called the replication layer which is a GDB in RAM like Redis cache DB that incrementally persists in backup storage for things not active in the game at the time like a player that is logged off. The servers are using rDMA (remote direct memory access) to read/write from the replication layer cache to stream in or persist anything that happens in the area of the universe map that server manages. In data centers servers have things called DPUs which are processor cards much faster than ethernet network cards and essentially can join the PCI bus of server motherboards together so they can share their resources at PCI bus speed and what makes 'cloud computing' and auto scaling servers on demand possible to power the web. They are tryng to get that tech to work at the demands a game server with real-time combat needs that can self-heal and auto scale to demand. Neo4j GDB is fast because most transactions are 1 index lookup to find your starting node on the graph then it walks/scans the graph from there. Each node and relationship record stores the pointers in RAM/on disk to their directly connected parent/children records. There is also no row padding RDMS eats to make fixed rows for its storage/search algorithms. Neo only stores the data that exists in each node/edge for smaller file sizes. Taiwan Telecom uses Neo and has over 1.2 billion node records and 10 billion relationships connecting the nodes and average search time returns results within 250ms or less and drives Realtime systems in their microservice architecture. An update 10 months later: We have had a couple live play demos in game on the new server meshing tech and it worked. We went from our classic server running an entire solar system for 100 players on one server instance to splitting the same solar system up across 4 server instances for 500 player capacity fairly smoothly. We are testing 6 servers for 1000 players now, but it degrades in performance due to other services they are still updating to handle the new server messaging system that gets backed up. It's not ready for live but its playable and technically works sans bugs they use us as live load testers playing the game as we normally would to find. We may have a long-term demo server running by Xmas and go live early next year on the new tech if all goes well.

[@criticalchai]

The big worry i had was latency. But then I thought if they cluster the various servers running the replication layer and shards (assuming shards would take on the role of each hosting a couple of containers) that moving from one shard to the next shard should be pretty fast and have pretty low latency between each other even if they have high latency between the servers and us. they may have to figure out clusters and probably if enough people showed up in one spot it would leak outside of the clusters to find any available shard to host containers. I'm curious how well it will deal with a ship fill of loot from bunkers when it transitions between servers/shards

[@miinyoo]

Yeah that's the real pickle. How does one do that with so many children? It seems possible but when the electrons hit the wires... Like FD mentioned. Protocol is everything if it is to be playable. Simple compressible and elegant.

[@Knight3rrant]

Not really. The entity within entity within entity capability of the graph database means you just move the shi entity and the associated entities within it go along. This can work for the ship entity as easily as a container entity.

[@TheKingoftheriff]

"lightning bolt, lightning bolt, lightning bolt" Fucking subscribed.

[@consumablecorner150]

Firs time watcher. "Perfect is the enemy of the good." Never heard that before! "..the truth will set you free." Definitely an empowering statement. You're a very soft-spoken bro. In fact, so much so that I could hardly hear you xD (edit: at least in the beginning)

[@JohnZingTTV]

this video convinced me to install the free 14 day epo install and try this game even with my poopy 4gb 1050 gtx ti gfx card keep up the content i think the 20k views show there is a high demand for this content and the format it was very real i look forward to see whatelse you will cover as far as other games are concerned. well next convention if yougain alot of traction on youtube mabey they will pay all exspenses so you can go taere nextime to citcon right? :D

[@IronPhysik]

I have a feeling that you may have misunderstood the job of the replication layer in all of this the way I myself understand it is that the RL is there to replicate objects for other servers on other zones so that each server can do the physics calculations for its own zone of objects, and it just displays the results of what the others did so its "replicates" objects for the player, and thats how you can shoot across boundries.

[@PlumHeadLJ]

51:20. I think there is no bullets, just raycasting outputs from player entity with saving the angle (vector) when leaving one zone and receiving it by another zone

[@TheKingoftheriff]

You also need a timestamp and an owner, bc the player isn't on the second server.

[@Nemethon]

I think it is the most important part of the whole project. :)

[@moonryder203]

We are all waiting for this tech to hit the live servers if they can pull this off. I have never been so addicted to a game ever in my life, this game is like every childhood dream I had of being able to do in a game and then some. 😂 This attempt to do this in a game will be historic for many in this industry.

[@TheNerd]

The prinicples oft this technology could change how we think about multiplayer.

[@Fluke2SS]

Overwatch is a bad fucking example, because despite the fact Blizzard said the principle of "Favor the shooter" is in place, what they don't tell you is the server tick rate is 60 or 64hz. So if you are running 120 FPS constant and you take a shot, you miss because the server (aka Replication Layer) misses 1/2 of the information the client sees. They recently changed Sombra's translocator ability which is directly effected by server tick rates. You can do a 180 degree turn throw the translocator, and the "Replication Layer" only sees you turning 90 degrees and as a result your translocator hits the wall next to you instead of where you were actually aiming on the client side. This is the same reason people claim to be behind a wall or corner and they die in a different location than what they see on their client. TL:DR Overwatch needs a 120hz+ server tick rate to make the game playable for most clients considering how extremely low the hardware requirement to play Overwatch actually is. CIG might be able to get away with this issue at 60hz tick rate because the game requires beast hardware to run. And it is not even optimized yet.

[@Whoadayson]

Cool video, Thx! CIG is doing big things, but I still have my skepticism that they are as far as they want us to think! Sidenote: dont worry about background noise. Its narely noticeable!

[@robmulally]

So interesting you mentioned the interaction at a distance, this was solved by another massive multiplayer game by making ordinanace that could possibly traverse withotu the player like large bombs or whatever, ownership was passed over to the server, and thereforce would just act in CIG case like a server owned object passing through each zone. This also allows objects that impact large numbers of people independant of the player, this is also basicalyl how i assume a player leavin their ship in an are would be handled if player despawned in combat etc, so I think this will be built in by default.

[@cryingsurrogate]

A minecraft mod called "Create" pulled off a similar thing with trains - travelling through unloaded world chunks. Since the rails are curves, it can follow them regardless of the chunk state. I'm guessing this pre-calculated curve approach would be the type of implementation for this game, since bullet distances are finite.

[@seanc6754]

I been a backer for about 4 years and I'm with you buddy i have a potentially unhealthy obsession with this game myself lol..even when i take a break from playing i watch a LOT of sc stuff lol

[@Alundil_55]

Thank you and well done

[@jasiekthejester7764]

Awesome dose of information! Thanks!

[@Shawnsrumi]

As SC backing I appreciate your perspective…thanks for the insight

[@kiblams]

The room used in this was is just for demonstration. I doubt they will use small areas per server in end use. They are more likely to have a city as one server and maybe the inside of a large ship being one server. So movement between servers wouldn't happen as often as you are assuming.

[@Antici---pation]

They have stated that the meshing is intended to be dynamic based on load in the future. You may be correct that the initial implementation might not have this issue, but they have set themselves up to need to tackle this issue in the future. An example they have given in the past is a single ship, like an Idris, having multiple dynamic meshes. "With Dynamic Server Meshing, it’s possible that large ships such as a Javelin could have their own dedicated server assigned to run the authoritative simulation for that ship and everything on it. However, we’re trying to avoid having inflexible rules about how entities get assigned to processing resources, so that might not always be the case. It comes down to efficiency in terms of both processing speed and server costs. If we had a hard rule that each Javelin and everything in it gets its own server, then it wouldn’t be very cost-efficient when a Javelin only has a handful of players on it. The same rule also wouldn’t be efficient in terms of server processing speed if there were hundreds of players all crowded into the same Javelin, as the rule would prevent us from distributing the processing load across multiple servers." robertsspaceindustries.com/comm-link/transmission/18397-Server-Meshing-And-Persistent-Streaming-Q-A

[@JC-Alan]

Great video brotha. Dont worry about background noise :) not much you can do abiut a hungry roommate. It's good to watch this after attending CitCon with my buds that arent in tech. Im neither a game dev or a regular developer, but as a systems admin I have some familiarity with data systems. Your comment about localizing the inter-server communication to something like a kubernetes cluster is spot on, if i had to guess. This falls in line with their initial idea of geographic shards (e.g. a shard for NA, SA, Europe, and Asia, etc), but that's gonna be a huge undertaking if they want the solution in finality to have everyone on a single shard. The laws of physics work against you there, so im very curious to see what directions they take with minimizing the latency on handoffs.

[@sc_cintara]

We are already able to play in distant regions if we like (you can select region manually when joining the game) and it works, though admittedly 0.5s ping is a bit "bouncy" 🙂. Nevertheless, unless you are doing combat, it is usually fine. An issue with shards is that they are so isolated from each other. You will never meet people who play on other shards. This is an issue for large global orgs with people wanting to play in their local shard, yet hang out with org members from other parts of the world. One solution could be to take advantage of the Amazon infrastructure and high performance networks between their DCs to set up a "follow the sun"-shard. So, if you log in to the "follow the sun"-shard, you will be in the region that is currently in it's prime time, but as the 24 hours move on, your server gets migrated to new DCs around the globe. There might also be local shards, for those that always want perfect ping, but if you are OK with high ping during off-peak time of day, you could make your home in the "follow the sun"-shard.

[@qdpqbp]

my english not that good you said youy make perfect the enemy of the good? if so you just described my life

[@free-dissociation]

haha mine too, all too often ;)

[@matthewfletcher4627]

The fact they are persisting entities across servers, is just a showcase of this... In reality... A server is going to swathe a HUGE span of the system... IF and the IF is pretty big here... players find themself on the cusp of a server boundry, yeah... Might be difficult... But as that could be the edge of a bubble that's millions of meter cubed.. Chances are slim... Getting to the edge of that bubble would mean you dropping out of QT at the right time

[@freelancerthe2561]

bold of you to assume theres not a lot of dead space in between, in spatial terms. One of their main goals with meshing and containers is to not manage space with nothing in it. The other half of that is to abstract out large chunks of the NPC simulation, so you don't need to spend a lot of server resources on them when there are no players to observe them. Its not a fixed number of bubbles across space, but a bunch of small roaming bubbles that merge and break off as things move around.

[@matthewfletcher4627]

@@freelancerthe2561 I don't think there will be... All that dead space will be on its own shard...

[@MaticTheProto]

@@freelancerthe2561dead space coincidentally is very easy to calculate as there is nothing there

[@mobiuscoreindustries]

Right now the edges are going to be fixed. Essentially they will delimit areas of big traffic and assign servers to each. But the goal is dynamic server meshing, meaning server authority changes based on the actual load demand. So that if a big chunk of load moves from one stellar area to another then servers will gradually cluster up in that area to divide the load. Potentially up to the divide happening within visual contact. Everything else that does not need to be seen is streamed out, and handed to the background simulation, which is an entirely separate thing that feeds to and from the in game simulation.

[@Sarsour_]

Awesome content!

[@strubbleler]

shiny man is SHINY ! love to see new people get into the game, and be amazed just as i was, even though im not a developer! love the yellow wall by the way!

[@nagasadhu3286]

FD's a 2014 backer ;)

[@marclamy364]

Players, just like servers, have a streaming container around them. It is likely that servers only need to see around them to facilitate simulation and transitions, but players have variable needs. For instance, you'd expect a player flying at 1200m/s to need a much bigger container to stream entities early enough to avoid popping. So to answer your question, if all 3 servers in this demo were in a straight line (purple, green, red) and bullets were physicalized, you could imagine a player in server purple shooting at a player in server red. The server purple would manage the physicalized bullet for a while, then server green and then red, potentially handling the collision with another player there. While the shooting player needs to see the result of their shot from server purple to red, server purple doesn't care. So they'd have very different streaming containers.

[@sevilnatas]

When he had the graph representation up, I don't remember seeing a bunch entities being spawned off the zone node, that you'd expect to see if they were physicalized entities, like other entities. Wonder how they are handling that?

[@free-dissociation]

Other commenters tell me that CIG moved from physicalized bullets back to hit scanning.

[@sevilnatas]

@@free-dissociation Ah, makes sense.

[@sc_cintara]

@@free-dissociation I don't think they are doing hit-scanning in the immediate-hit-where-you-aim-sense. I think they are doing bullets on the GPU. At the least for ship combat there is a hard requirement to follow where the shots are moving (those beautiful arcs of fire between ships), so for large scale fleet combat they still need to move bullets between zones. However, I have some vague memory that even for ground, they still trace bullets through the air, they just have some GPU trick to draw them very quickly rather than traditional geometry. BTW, this is easily testable with a sniper rifle.

[@NoNaNelson69]

transition is most likely when its almost done- if you look at armistice zones, its basically the same how to server detects once ur in or out you enter when you are fully in and leave when fully out

[@DaeDroug]

My impression was that the red, green, and purple servers where running on 3 different laptops for the demo. Which having it working like that rather than over whatever in rack inter server bus they might be able to utilize is especially impressive.

[@madsmith1352]

No, the system (whether a laptop or a more powerful full pc) is spawning three server processes running locally on the system. There’s no significant network latency being incorporated in this demo as none of the network communication packets need go out over the wire.

[@sodiufas]

@@madsmith1352 it shouldn't travel far in cloud too, why would it have to?

[@jackinthebox301]

As Madsmith said, in this demo you can see him click in his dev program and change how many servers are being created. The whole demo is being run off of one PC. It's also entirely reasonable to expect that their plan takes into account where the simulation servers are located in relation to the replication layer servers and the latency that will inevitably introduce. These are network engineers after all.

[@sodiufas]

​@@jackinthebox301 they are basically should be in the same datacenter. Why there will be latency, thats my point. U can go on their website and look where are they located on the map.

[@sodiufas]

it was not, they had servers at stage, but no way laptop could run it@@madsmith1352

[@LumiLupo]

you had 4 minutes of talk before the actual content of the video started, i'd recommend you to keep that a little shorter (cuts do help) so people keep watching. or you use timestamps so people can jump directly to the mainpart what means a lot of people will not see what had come before ^^

[@marclamy364]

You're correct about the fact that running all servers and the replication layer on the same machine, it eliminates the latency issue. That said, the latency between game servers and the replication layer, assuming running on the same data center, should be negligible. Assuming the game servers simulations run at approximately 60 ticks per second, that gives around 16ms per tick. With an average network latency of, say, 1ms within the same data center, and standard client-server synchronization techniques applied (client prediction, server reconciliation), it is very possible that no latency would be visible in this very simple demo.

[@free-dissociation]

Wellllllll. It doesn't _eliminate_ the latency issues, unfortunately-I wish it did. There's all kinds of nonsense that gets in the way. (Whatever nonsense the scheduler has decided it's getting up to that day is the one that I remember biting me personally.) But I'll certainly take same-machine latency over even same-datacenter latency.

[@eltreum1]

Message from the future. It worked as a drop-in replacement for our classic 1 server for 100 players in 1 solar system like we have now. So far in upscaling testing one solar system across 4 servers with 500 players in live play load testing is working well. Testing 6 severs across a solar system for 1000 player capacity degrades but basically works. They are re-writing their other backend services to use the new messaging system and replication layer to keep up with the load as the old messaging system was getting backed up. Net positive progress so far. I think our GPUs will bottleneck before the player count on a server does based on the 500 player test when we all gathered our ships at one location at a city spaceport.

[@sxnorthrop]

Problem 1: I suspect that there's additional communication between each of the adjacent servers that allows them to notify their neighbors of any raycasts that may potentially penetrate an entity not under their authority. The question is what happens when there's latency between each server? What if the bullet has a large enough range to go through, say, 3 different servers? That would make the experience even *worse* than using a single server because you'd be tripling the latency cost. They'd definitely need to avoid placing many of these zones in wide open areas. Problem 2: If there are players that know the boundaries of each of these servers, what happens if that group all decides in a single moment to move between each one rapidly? Depending on how computationally expensive the "trade off" algorithm is a large enough group of players could potentially cause the replication server to lag or crash outright (or even cause a nasty deadlock depending on how they have things set up). Problem 3: This is a space simulator that allows you to travel hundreds of times faster than the speed of light. What happens when player 1 leaves server A and transitions to server B, but before the replication server is able to handle that and server B knows that it has player 1, player 1 is now transitioning from server B (at least it's zone), to server C, but it's still under server A's authority? Does server A still claim to have player 1 and so update the transaction to say server C instead of B? I'm sure there's something that could blow up there in the wrong circumstances, or potentially a teleportation exploit. Maybe I'm mistaken. As with all things new, something's bound to break! But G** bless them for being the first ones to try!

[@free-dissociation]

> The question is what happens when there's latency between each server? And the first rule of distributed systems is, there's *always* latency between servers. > If there are players that know the boundaries of each of these servers, what happens if that group all decides in a single moment to move between each one rapidly? This is a pretty common tactic that EVE Online corps have used to turn battles in their favor, from the launch of the game, so yeah-a big problem, and one CIG will have to address. > What happens when player 1 leaves server A and transitions to server B, but before the replication server is able to handle that and server B knows that it has player 1, player 1 is now transitioning from server B (at least it's zone), to server C, but it's still under server A's authority? Probably, the client is not allowed to initiate the transition to server C before the server acknowledges the authority transfer to server B. Hope they tested the heck out of that state machine!

[@matthewvelazquez2013]

So, uh, The inference I'm making at minute 46 Is that the same concepts behind A.I. frame generation are being applied to entities transitioning from one zone to another? That's why zone adjacency must be so important? The line of demarcation between zones Constitutes a 'generated frame.' Based on the mock up on the screen, A true frame could be symbolized As the purple zone. The generated frame could be symbolized as the green zone. And the following true frame could be symbolized as The red zone. Scale this down And compress the operation inside a single line of demarcation Between adjacent zones, right?

[@surject]

There are a few glitches in the demo, for example the 'box' in the middle left in the green zone (bottom right view) moves at 58:34 (player in the first person view shoots it) but it does not move in the fpv nor the other 2 views.. and then it suddenly re-appears again in the bottom right view at 1:00:31 - hmm?

[@RanmaKei]

I see that too and may be a bug. They did just put this all together as the first iteration of server meshing so I'd imagine they need to tweak the code to get it perfectly in sync with all the different layers of the simulation. I'd imagine you would need to do some complex checks and those are done on the server side as opposed to the client. These checks will likely be involved with latency between servers as well. It could also improve with time as they work out the tech. It's not so much about running into issues with simulation as it is probably more about how you handle those issues since you will have latency, memory pressure, CPU throttle, crashes, etc. that you have to account for in code at replication layer and simulation layer. Will just have to wait for the next iteration of the tech and see how dialed in it gets. As it stands its very impressive that they have it working to this degree.

[@free-dissociation]

ooh hmm. thanks for pointing those out. I'll have to go over this and see if I think it tells me anything more about there implementation.

[@HitmannDDD]

We saw a demonstration that carved up the game space by 3d spacial boundaries because it's easier to visualize in a demo. However, given what you know about the tech, what additional technical challenges do see regarding removal of the spacial boundries for servers? The use case being that multiple servers could then handle a single 3d gamespace, thus potentially increasing the max player count in close proximity.

[@josephbaker6461]

The bullets in Star Citizen are indeed individually modeled, probably for the reasons he said. If you play with the active camera in game you can see it.

[@sevilnatas]

I wonder what type of message bus tech they are using. It's got to be fast and I wonder if it is a guaranteed delivery system or they have room for lost messages.

[@josiah9388]

It would be interesting know when a 30k Happens is the crash a single service on a server or the physical server. That would drastically change the tech. If it's a service or VM that crashes you have the replication layer on the same physical hardware. If the replication layer is on different hardware the transitions will be much more impressive. not to mention whatever tech they have for spinning up new "Servers" as needed.

[@free-dissociation]

It's just the game server service process crashing, it's not the underlying hardware. (Think how much that would have cost CIG back when we were getting 30ks every day in game if it were the hardware ;)

[@josiah9388]

@free-dissociation I agree that's how it works currently. But it may not be in full server meshing.

[@steved4475]

They need cross multiple server interactions to work. Shoot from purple past green and into red. That would be the case of Shooting out the MSR cargo bay through space and into the Idris hanger to hit a target.

[@mobiuscoreindustries]

It's likely that server boundaries will greatly extend past any individual interaction so that in that particular described case all entities would always be on the same server. To the server however at a macro scale all information is the same. So it may not need to account for someone shooting across the boundary but it may encounter hundreds of ships, their cargo and items crossing the boundary, all the while the players inside are actively doing things. Not counting NPCs doing the exact same.

[@sc_cintara]

@@mobiuscoreindustries You are right, but only in the first step, static server meshing. Once they get dynamic server meshing (which they have to do since they hope to have a single shard per geographic region), they have to split up the tens of thousands of players per geographic region into their own 100-person (or whatever) servers. That means that the server boundaries will be really tight. They need to expand to many more landing zones too, or it will be unbearable to go shopping! 😂

[@Think666_]

For those who don't speak Engineer "brave choice" = "bad choice" unless you know something they don't...

[@craigwoodward8455]

7:16 CIG dont have a habit of avoiding bugs during live demos... where'd you get that? It's on the bingo for one of their live demos to crash. XD It almost always messes up for them, the amount of times its crashed entirely during the demo has been a meme. lol

[@ParaquatSC]

I also think without a doubt that the server zone will be a concentric area within another server. That way there will only ever be one or two "borders". So Stanton as a massive playzone, then a zone for each planetary system, then perhaps a zone for the main city, just as an example.

[@sodiufas]

Oh ok, another thing about SC is projectiles physicalized, that's why we doesn't have any hit scan weapons.

[@EdsEnemy]

i don't have time to listen to this right now but chapters or timestamps in the description for the main points you wish to make would help a lot

[@VectorParalax]

I think you just became a Star citizen youtuber!

[@absentminded02]

The statement i heard from people at the event was Chris called the team cowards if they didnt do this live. The fact that it worked at all is really promising

[@sodiufas]

bullshit

[@absentminded02]

@48:30 I want to point out that, He's not running all three servers on a single laptop, There are 3 laptops back stage that are running the Servers. This was clarified by people who were at the Event talking with the engineers after the panel.

[@sodiufas]

​@@absentminded02 yep i agree pretty much

[@Crispy_Steak]

one of the servers locally was on that machine for certain, it bound to localhost in the logs when starting up, failing to bind to at least 2 ports (12300 and 12301) at least when the 1st instance booted. I'm pretty dang sure this on the machine on stage, the 3 dgs mesh instances use just under a 5 GB memory each at the start with the minimal environment being loaded. Technically they probably could have, but having a bunch of weird machines running a demo wouldn't be something I would be comfortable doing on stage.

[@sketchtherapy1218]

Would you please do a video with Mr Tybio about the 5 toughest problems in computer science CIG is taking on or 1 video for each of the 5 with Mr Tybio?

[@matthewvelazquez2013]

So, uh, At minute 35 the inference I'm making is That the client can crash his game By spamming The activation and deactivation of possible In-game surveillance drones. Maybe the simple solution is to insert a card Displaying the words acquiring signal In the mobi glass of the player character - Giving the game engine actual time to call on everything contained in non adjacent locations? Would not the spawning of a possible surveillance drone Initiate the same rules used for a player character? Could a nefarious user then rapidly spawn The multiplication of in-game surveillance drones To crash the server?

[@rakaydosdraj8405]

A drone with a camera doesnt need to see anything unless it's being operated, which means you can stow the operator's perspective and jump to the drone's "remove view" perspective, which takes up the entire FOV. You dont need to have both in operation at once, which limits the number of observers in the system to the number of players.

[@matthewvelazquez2013]

@@rakaydosdraj8405 thank you for the specific insight. But, this IS Star Citizen so maybe Chris has 2 Savant developers working together in a 'Hidey Hole' somewhere to bring 'picture-in-picture' surveillance drone operation into the mobi glass interface? Heh-heh-heh. Not implausible, considering what they've been able to do so far...Ha ha ha ha...

[@rakaydosdraj8405]

@@matthewvelazquez2013 It IS star citizen, so Remote Turrets already exist as a thing, in live patch. the "full field of view" workaround seems to be what they're going with. Generalizing that system to recon drones is a cleaner solution.

[@matthewvelazquez2013]

@@rakaydosdraj8405 thanks.

[@pale0wl]

A mounted SHURE SM7B will help with your roommate situation. You will want to set a gate though. You can gate through OBS but having a gate on the audio interface is nicer for monitoring

[@kemo_l7350]

How could I get a job with cloud imperium games?

[@kemo_l7350]

I mean that I don't have that many skills and I am in Jordan so the only place I can can learn from is the internet

[@gorganhorn6872]

Please tell me your uncle is Jason Issacs, because that would make perfect sense in this reality. BTW thanks for your expertise and taking the time to unpack all the details that even CIG's show cannot do. +1 sub

[@carllong8954]

"distributed system is the hard part" That's why we have all been waiting for this game for 12 years now. I can remember way back at the beginning when CR was talking about having 2 cap ships fighting and each was a server and shooting from one ship to the other would involved hand offs between 3 different servers at the shot moves. That is why this demo might be the biggest step in the development of this game. This demo showed everyone who understood and has been waiting that it may actually be possible.

[@unknowncitizen9300]

Can someone give an idea, at what point the replication layer could struggle updating the entity graph with the appropriate tick rate (lets say 30)? Because for my understanding, if meshing would work as intended, this would become the next bottleneck, or am I wrong here. As I don't have a great understanding of game engines and server tech, I don't really know if like the entity graph update is 100 times less compute intense compared to the engine work (physics etc). Hope this isn't a stupid question...

[@free-dissociation]

You're right, this *is* gonna be a limitation, and it's gonna depend a lot on specific implementation details. I unfortunately don't feel like i have a good intuitive sense of where and when it's likely to bottleneck. Updating the entity graph on a transaction-by-transaction basis is definitely less compute intensive than the physics simulation, but because the global entity graph has to be able to receive and process updates from *everybody* currently playing, not just the players in one region of one shard, it's still a hard problem. Really the best practice with optimization stuff like this in computing is just "try it and find out."

[@sc_cintara]

@@free-dissociation I think the answer to this is that the entity graph is not the replication layer. The replication layer might be caching things for the entity graph. The replication layer is also redundant and horizontally scalable, and the entity graph is the persistent storage. As long as the replication layer is able to scale horizontally, it can take the load for tens of thousands of simultaneuous players in a single shard.

[@Crispy_Steak]

the replication layer uses a queue to update entity graph I think, per 2021 cit con presentation at least, shoudn't have a fixed (30 ticks per second) update rate that way. Your other question I don't know enough about how this works to answer.

[@adamchambers1393]

They have had bugs on every demo they have done for their live stuff tbh. Their video stuff course not so much. But even the pre-recorded video for Star Engine was buggy which why they been fixing it since :)

[@Crittek]

The replication layer is also moving the client using its input. Most of their idea is ripped out of Unreal Engines networking, everything is server authoritative. As a consequence, what this does and nobody is talking about is enabling very effective anti-cheat measures.

[@thepurpleperp4564]

I would expect that the majority of the boundaries between servers will be out in empty space, not in the middle of an area where high action fps pvp is generally happening, no? Not only does it reduce the number of cross-overs at any given time but there will rarely be anything complicated or important happening at these boundaries. Probably only quantum travel. EDIT: Also, isn't it possible that there's never a situation where you could even shoot across 2 servers?

[@free-dissociation]

It *is* possible to set this up such that there's never a situation where you could shoot across two servers, so if they're demo'ing that ability then they must expect it to matter.

[@BroodlingX2]

There ultimate plan is to have large ships with 50 + crew be its own server. Basically a server inside of a server. So projectiles would need to be able to cross servers since every time you shoot at a large ship it will need to go into another server.

[@thepurpleperp4564]

@@BroodlingX2 Roger that, I was talking more about shooting from one server, through another server, and into a third. Addressing the L-shaped demo.

[@80KBM]

Thankyou. I´ve been waiting for someone that can explain this tech in an "easy" way. A question. The scope is that this type of tech will make it possible (they say) to have (lets go a bit extreme) 5000 people on a planet surfice, and lets say 10 capital ships with 20 people inside each, and flying in low planet orbit. shooting, debris falling down and explode on the groundtroops. And all the broken ships, loot, bodies will be there for lets say"a year" for everyone to go looting, salvaging. Do you think this tech is the "answer" to make that giant type gameplay possible?

[@free-dissociation]

That's the goal, and if they're going to pull it off, something shaped like this is what will enable that. Whether they can pull it off, at a cost they can afford-we'll find out! It certainly looks like they're on the right track at least.

[@sevilnatas]

SeconfLife has been doing this for a very long time. Not saying that they do it better or that their engine would be up to running such a complex game like Star Citizen, but there is something to be said that they had this distributed model running more than a decade ago. I wonder how the two solutions compare?

[@CSKefka]

They do not.

[@sevilnatas]

@@CSKefka They do to.

[@free-dissociation]

Yeah, EVE Online as well. I know a tiny bit about EVE's internals and nothing about Second Life's and would love to talk with people who know them better. EVE at least does handoff between servers at the solar system level, and it's pretty clear to me that CIG want and need something much more fine-grained here.

[@sevilnatas]

@@free-dissociation Yes, definitely finer grain for CIG. From wikipedia "Each full region (an area of 256×256 meters) in the Second Life "grid" runs on a single dedicated core of a multi-core server." I was originally thinking they were doing a region per server instance, but it looks like it is a region per core. Interesting. So the big explosion of core per cpu, probably really benefitted them. I have a dual cpu EPYC server running in my rack at home that has 48 cores total, that I pieced together off of eBay from Chinese decommissioned hardware for like $1k. It is like 3 generations old EPYCs and used RAM, but it runs like a champ and I guess it could run 48 regions of SecondLife, crazy!. I imagine their server footprint, to support the same number of users, gets smaller every year.

[@free-dissociation]

​@@sevilnatas And in this day and age individual cores have gotten so performant that it's entirely possible they could run multiple server processes per core. That was one of my surprises from the interview I did with Zach Johnson-he said that Diablo II Resurrected was able to run that way: kzbin.info/www/bejne/i6bTgal-pMiIh8k

[@ThorbjrnPrytz]

They run on Amazon cloud servers, And the internal high speed network on the datacenter should minimize latency between servers

[@Yokeus]

I would be curious with how big CIG decides to make these zones. If the red zone is 1 Million meters across for a whole planet, you wouldn't be able to shoot across three zones like you had mentioned. The weapons and scanning don't have those kind of ranges at the moment.

[@doubledigital_]

the solar system its self is a giant zone, they can nestle everything .. they just need to link them all to servers now.. fiddly but its more or less done :))

[@realmwatters2977]

We are star citizen players and we want your opinion on server meshing. 07 see you in verse!

[@a__random__person]

in future, since you're recording a youtube video in 4k.. you should either upload in 4k as well so we can read the text in the video easier.. otherwise reduce the source video to 1080p :)

[@St4rQu3st]

You could pool objects such as bullets so they are just reusing bullets instead of creating and replicating them across. Just pool and use them and send them back to the pool.

[@AuthenticGadzooks]

I'm not really surprised that it works in a test setup. You can get anything to work in a test setup. What I'm wondering is whether it will keep working when this gets upscaled to thousands of simultaneous players and millions of individual objects being streamed between different clusters and the interactions between all those entities. Time will tell...

[@theamericanaromantic]

Main things I took away from this video: 1) "Distributed systems are *really* hard" 2) CIG is in charge of creating and maintaining said distributed system 3) We're fucked.

[@DerKilianist]

I only skipped through the video, why the 3rd point?

[@theamericanaromantic]

@@DerKilianist Just a little joke because well… it’s CIG. But honestly I’m skeptically optimistic CIG can pull it off.

[@DerKilianist]

@@theamericanaromantic I hope they can. Just wanna play this fuckin game

[@Martin-sx4bx]

@theamericanaromantic CIG has very talented peoples

[@dawnphantom2439]

3) Were not "fucked" because they tested it at scale yesterday with Evocati. It worked.

[@surslaughter3524]

Great video. I have one issue though. Saying CR didn’t risk anything when speaking about anything in SC is wrong. He put everything on the line. From most of the gaming community screaming scam for 10 years. Half of the SC community doing the same. Every issue, every marketing disaster. Every failed patch. Every projected release date for ships/features/patches. He put his name and career on the line every day for 10 years. He deserves some credit.

[@seveneternal7988]

"U didnt have all the pain, im sorry" To be honest you should show a bit more of respectoto CR. HE is himself programer and he did himself a large portion of the Zone System which is the fundamental piece of tech in which this is holded on, and the only rock solid thech in the entire SC world.. (and he worked on several other pieces of thech in SC, I just named one because its mentioned a lot in this video) You should really see the 2015 ish interviews of CR in several youtube channels, I remeber one in gammer nexus and such, and he going REALLY into technical details on how he (and they) did the Zone Sytem. I can tell you that he went way more technical than u did in your videos because he knows what he is talking about. Lets also not forget that he was programing and doing games even before game engines existed.

[@doubledigital_]

player can see it because the replication layer is now showing all clients all the logic ,, but the server its self has streamed out the data.. saving a fuck load of memory the replication layer

[@AlbertoMartinez765]

Its Nice to hear from someone who is actually Knowledge about the subject.

[@joshoowa]

Was this live? I'd be interested to watch the show live with you on twitch or youtube or something in the future.

[@thePrisoner1000]

Yes, it was done live at the event.

[@pylotlight]

@@thePrisoner1000 not what he was asking.....

[@80KBM]

I think it is about 10 hours total. I cant find one thing they talked about that i thought was "bad" or "missleading" And i am a bit picky.

[@Sypher474]

The presentation was live on Twitch originally then uploaded to YT, this reaction was a recorded video.

[@Austintatious_One]

I think alot of changes of aproch over the years have come from a shaky architecture.. Originally I think the engine and data sets where being optimised for optane, possibly with SPDK.. Now though I think they have had to migrate to CXL, this will possibly be better latency and scale wise, definitely it will be better for support and implementation going forward though.. 👌 Also.. Shooter calls the shot I don't think will be relevant as the shots are raycast and outcome it self is authoritive and know instantly by both parties.. 🤔 (Will still be a micro second faster for the party taking the shot but not in any relevant way compared to older z axis rounding calculations which does have its inherent issues in group play)

[@Quent1nB]

My thoughts and question: - bullets: while they are going through the RL so that other servers see it, I doubt they are going to the PES graph db (= no write spam), I assume they ban some entities that have a short ttl and no purpose. - your comment about latency on real servers is true, however a DGS will usually serve maybe a system, a planet, or a city. Cases where it will be assigned to a space ship will probably be rare, and I doubt they will go "lower" than that, so we will rarely observe any latency from that. If it comes to indeed splitting ships/UGFs into multiple servers, I'm sure there are some solution (your idea with k8s pods on the same machine is a good one tbh) - question: when sending objects between two adjacent DGS, I understand that both have a copy of the object, and only one has authority. Do you think both DGS run the simulation for that object though, when they are close to the "border" at least, so that handing over the authority is as smooth as possible? Thank you for your videos!

[@free-dissociation]

> - bullets: while they are going through the RL so that other servers see it, I doubt they are going to the PES graph db (= no write spam), I assume they ban some entities that have a short ttl and no purpose. Yeah agreed that bullets aren't going to go back to the global entity database. Even there though the write spam on the replication layer strikes me as high, and my read is that they *have* to go through the replication layer. It's possible CIG have worked out some hack with raycasting as other commenters here have suggested. > - question: when sending objects between two adjacent DGS, I understand that both have a copy of the object, and only one has authority. Do you think both DGS run the simulation for that object though, when they are close to the "border" at least, so that handing over the authority is as smooth as possible? I think naïvely I would try without this, just doing the handoff between physics simulations when the authority gets handed off, to simplify the process. I can imagine there being enough randomness in certain kinds of physics simulations that two servers might simulate the same object very differently, but there I think the better solution is to make the physics simulation behave as close to identically across all servers by e.g. stuffing the necessary random seed into the object data.

[@Quent1nB]

@@free-dissociation Oh good point about the randomness of physics sim. I guess we'll never know 100% how things work unless someone from cig tells us, but I would love to have all these details. It sure is fascinating to watch your channel and see how someone else thinks. I work in IT but nothing to do with gaming.

[@j.d.4697]

Opinions from people in the industry on SC are rare but we sure would like to know what people think other than CIG themselves or all those people who are 100% opinion and 0% knowledge.

[@BradAhrens]

Subbed because I wanna see you lose your damn mind when it's in-game! 😂 GO, CIG!

[@sodiufas]

Have u seen a comment from CIG, where they stated they are in touch with people actually making this DB and they too are taking the challenge? Something will grow out of here atleast.

[@free-dissociation]

Yeah Chris Roberts's comment mentioned that they had the graph database engineers working with them on getting the bug they were experiencing fixed.

[@IntegralTriangle]

bullets are physicalized, yes.

[@czbombah]

Security IT engineer talks about game :D

[@mr.potato9449]

As someone who doesn't know much about programming, what does this mean in regards to server player limits? Say the server limit is 100 and call this demo a city. Without meshing you could only have 100 people in this city so would only see and interact with 99 people. From what I gather of it, with the city now being 3 zones and a different server having authority over each zone then you could have 90 people in the purple zone and 60 in the green zone so if you were in the purple you would now be able to see and interact with 149 people and not just the 99 you could when it was just 1 server with no meshing. Is that correct or am I way off? If that is correct then the servers still have the 100 limit so what happens if there are 100 people in the green zone and you move from the purple zone to the green zone?

[@free-dissociation]

You're correct! The goal here is to be able to substantially increase the player count on what we've called "servers" up to this point-what CIG are now calling it a "shard", and it is made up of multiple servers. Ideally in the case you describe you can see and interact with all 200 people (100 in the green zone and 100 in the purple zone, each on their own server) seamlessly as though they were all on a single server.

[@mr.potato9449]

@@free-dissociation So what will happen when you move into a zone which is already full? Is there some kind of dynamic system where it will split that full zone into smaller zones (which now have their own server and limit) until there server in control of the area your entering is no longer full? If something like that is the case I would think there would have to be some kind of noticeable latency with the amount of times they might have to move people from server to server if an area is full and after the split other people have already entered it making it full again and needing additional splitting and so on o.O

[@oskar1799]

Hey man, love the idea. I have some note´s, Keep it short prepare a script, use jump cut´s and dont stop to often about some small detail´s, try and translate the thing being shown into "Human language" so the normal person understands what is being shown and if it really impressive or just marketing. Also maybe expand from only SC and go into other game´s and it´s game engine feature. Keep it up

[@sc_cintara]

These are good suggestions, however I am not sure about the translation into "human language" bit. There are many youtubers that try to present these technical things in ways that everyone understands, there isn't really a need for another. This video is special because it doesn't do that, it digs deep into the details, stays fully technical and speaks to developers and engineers. That is valuable! One should always be as simple as possible, but never at the cost of fidelity. If you are afraid of mentioning something like PAXOS/RAFT, you can't get the full message across. So I think there is good value in keeping it fully technical, we have enough of those that speak to everyone. I would like to see a youtuber take the engineering part of the monthly progress report and talk about it in a technical way.

[@Whoisitthatknowsyourthinking]

we don't want them explained in basic terms dude. Most people watching this video want a high level explanation of what may or may not be happening between different services etc.. It's refreshing for me seeing as my line of work is Devops/AWS solutions architecture and I deal with things like (EC2, ALB's, Lambdas, RDS, mong DB , EKS, the list goes on ). I want to know at a high level how this is all implemented in the gamming side of things which I don't have a lot of experience with. I do get what you're saying but there are 100 videos that are layman's terms that basically sum everything up in a minute and a half with answers like "its the future ! you can have as many servers as you want ! thousands of people in one server ! It will change gaming forever! Capitol ship can be a server ! Magical tech that only CIG can do ! Ect , ect ... It's extremely important that we have people in the community with a very high level understanding of the kind of Network architecture / software it takes to do this, that way people that are less techniacally inclinded can manage there expectations a bit in terms of what all this tech can and can NOT do.

[@muf6861]

You don't worry about latency to much in a datacenter like Amazon between servers today if you pay to want this bandwith. Where I work our servers hand off highly active memory between servers when they switch hosts in the a network easily with no notice of latency really. As long as you keep the server in the same building or nearby buildings because distance with the speed of light that data travels is a thing. The real deal is the speed that the replication layer (very high speed database connection) service can handle all the traffic from all the static servers and in the future when server meshing and it will be object containers like outposts, cities, planets, ships or part of the solarsystem like shown in the demo, where it was one room divided into 3 object containers. But this tech I can't see why can't be scaled up as long as the replication layer can keep up. CIG really are doing what every MMO game company would dream of doing but won't invest the time or money on (we are talking minimum 5 years of R&D on a dedicated team at CIG and external database company). And also the bandwith/speed today in modern and good networks today in the top hosting centers are way faster than a single computers memory speed/bandwith.

[@free-dissociation]

> Where I work our servers hand off highly active memory between servers when they switch hosts in the a network easily with no notice of latency really. Interesting. Is this like a compute host which is shutting down transparently snapshotting a running VM and transferring the snapshot to another compute host where it starts up and seamlessly takes over without user-visible downtime? I know there's a lot of wild stuff that's possible at the hardware level, increasingly supercomputer level stuff, but I haven't gotten a good read how deep in bed CIG are with AWS to pull stuff like this off, or really how much they need to. And IME often the cloud providers _promise_ this but whether it _actually works_ is another question. And yeah, when I talk with folks at other MMOs companies it sounds like they haven't invested the R&D and are running stuff in paradigms that are 5-10 years behind current, so I'll be curious if CIG are really pushing the envelope like that here.