Development Update with Server Meshing Technology - 11AM PT Wednesday, July 3, 2024

[@MargaretKrohn]

Glorious Ashes of Creation Community, a HUGE thank you for continually tuning into our development updates. We are getting SOOOOO CLOSE to A2 coming your way! Our June showcase was the most technically oriented we’ve ever done, and it’s all in an effort to share with our glorious audience what exactly goes into making this MMO, and what is being tested in A2. We know these kinds of updates aren’t as exciting for some folks as the gameplay-focused ones, but they are important. We would love your feedback, and we plan on announcing the A2 start date at the end of this month! Also, we plan to begin sending out test email invites to some A1 users this month as noted on the stream, so stay tuned for more information!

[@AszkalonOfVerra]

We all assume dear Steven is now better again. 😎 Thanks very much for (finally ^.^) another Art Update. I myself am really addicted to see anything new from your awesome Fantasy Universe your MMO Game plays in. 🙂

[@rodney3297]

What a great job taking such a super techy topic and making it digestible to the masses. As a IT professional I can certainly tell you non tech people cannot entertain it for very long lol. Nothing can make your mother bored faster than taking about a motherboard.

[@josephcruz8797]

I wish you guys were on the East Coast

[@chutch1122]

As a software engineer, I have put a lot of thought into how I might design a dynamic gridding solution for multiplayer games (like MMOs). This stream was very interesting as well as reaffirming for me because the ways you are achieving scale are quite similar to my designs. I love the technical deep dive content like this every once in awhile. Keep it coming!

[@milesnapue]

1

[@Mr.Nubtastic]

Stop saying it wasn't exciting because it was tech. We gamer nerds loooove this stuff! Stream was definitely a 10/10 💪😎

[@AshesofCreation]

Awwwwww, thanks. Glad you thought it was exciting. We just know it's not everyone's cup of tea. ;)

[@leanture3685]

@@AshesofCreation it is mine! :D

[@darklink911]

@@AshesofCreation I'd say this was top 3 best dev streams and I think even for non-nerds it might have been more interesting than you think. I would say that apart from the deep dive into network engineering which the team and the presentation did a great job explaining in layman terms it shows something we all MMO gamers know. And that is laggy servers because a world boss is being killed or an event is starting somewhere in the world. What makes this dev stream so exciting apart from knowing how things work in the background is your problem solving, your goals and how you've taken on the challenges to solve them. You are not satisified creating a game with laggy server when world events happen like every other game has settled on and every MMO-gamer just has gotten used to without liking it. And you are pushing boundaries and tweaking the unreal engine to multi-threading just to get that effeciency to where you need it to be for your OWN standards and goals. That is VERY exciting to see because that means soo much more in the long run. It means you deeply care about delivering and improving the experience to something not seen and experienced before. Even after the game has released we you choose to play the game will see those improvements over the years to come after release if you keep the same attitude forward. That is what this stream also told us if you read between the lines.

[@indianakawa3752]

@@AshesofCreation Players who have played DAOC or Warhammer Online know how important this video is. Good gameplay without server/graphics stability is not good gameplay. Thank you for sharing.

[@tv.evostar]

100% agreed 👍

[@Nyce-Gaming]

All of you did a great job breaking down and explaining the most nerdy tech ever. Bravo for the awesome visual aids as well. Excited for good performance in large battles!

[@AshesofCreation]

D'aww, thanks! 😊

[@christ.4977]

This is great content! An MMO that isn't built as a distributed system and can't scale horizontally will always be gimped and immersion breaking. What you guys are doing is paving the way as the standard for all MMOs.

[@AshesofCreation]

💖

[@Artmos]

As a game dev doing both front end and back end stuff, I find videos like these very interesting. Good job

[@CaptainShack]

While I know Ashes is focused on a PvP. I wonder if we will see large scale PvE events. it be nice to see an MMO where the undead or ork horde is actually a threat to the world and not just waiting in a field to be farmed for XP.

[@swift4436]

Very much enjoyed you all sharing this with us folks :)

[@repsejnworb]

Saying it again, this and the stream where you talked about batching and generating materials on the fly and more to improve performance - are my favorite Development Updates!

[@Fantmx]

I appreciate the troll, Steve N. Great stream. I'm a tech nerd so this was fun!

[@AshesofCreation]

Haha! =P All in good friend homie.

[@MagicMan166]

Honestly loved hearing the detailed explanation!

[@a.plipps]

Absolutely loved the in depth technical discussion! Didnt understand much but loved listening to the difficult challenges the team overcame and did what sounded to be impossible! Keep breaking boundaries team!!! Also Im extremely excited to be able to purchase the A2 key cause im not a broke university student now! Hoping it'll be reasonably priced!

[@AshesofCreation]

YAY! Glad you enjoyed it!

[@rpgguy7429]

I really enjoyed the Tech discussion, even with little actual knowledge its really cool to take a look under the hood and admire the mass of moving parts and shed light on issues you guys are facing. It's been very informative. Thanks!

[@AshesofCreation]

You're welcome. Thank YOU for joining and following along with our development journey.

[@visionofpd]

This was bloody AMAZING!! One of the best streams honestly, loved it ⭐️

[@AshesofCreation]

Aw, so glad you enjoyed it! Thanks for the kind words and following along with our development journey.

[@UphillHook]

Was watching this whit my friend we had fun and learned something new

[@AshesofCreation]

🧠

[@Huskey789]

As someone that has recently started to learn about computer science, this was amazing stream. Thank you guys so much! Very inspiring!

[@vinnyjj12]

Coldest line from this video "I did some research on multithreading on unreal and everyone said it's impossible..... but we did it anyways"

[@Kezzic]

To whomever is handling the media on this channel, don't look down on the metrics- all this information is vital and needed for the community. Viewership will boom when the game is released.

[@DoingFavors]

Absolutely phenomenal video, I don't understand the complex parts, but I get how insanely complex it is to do. If it were easy, everyone else would be doing it. "But it's still not enough" hang onto that mantra

[@PowermanFR14]

Really loved the deep dive into server meshing, thanks!

[@UlfhednarTV]

Really enjoyed this type of content on the stream, amazing job!

[@AshesofCreation]

Thanks for tuning in!

[@c.wombatz]

This tech breakdown gave me more of a positive outlook toward AoC than the past year of other videos combined. That isn't to say I haven't loved all the other videos, but ones like this which dig into the hard challenges and how the team is overcoming them prove that development is moving in the right direction.

[@SurgeTheDrink]

@MargaretKrohn of alllllll people, you were the one who helped me get a PS2 Beta key. I conversed with you all those years ago... Seeing you here makes me happy that you are continuing to prosper. I look forward to seeing AoC getting more work done!

[@Gordojum]

This is incredible! Server meshing is one of the primary issues Star Citizen is working to implement as well. I'm glad to see this fantastic technical deep dive; clearly great talent at AoC!

[@lexavellino]

For what it's worth, this is actually a much more interesting topic to me than a lot of the standard content updates. My team is also building applications on Unreal Engine.

[@AurorJeffrey]

Hexagonal server authorities are mathematically more effective since you can still stitch with 100% coverage but only ever have a three server convergence. As opposed to square or rectangle where you have four server convergence at corners.

[@AurorJeffrey]

Think about it like heat transfer across mesh, there’s a reason why beehives are hexagonal not square

[@AurorJeffrey]

Although each server would face five distributed partner servers instead of three, it would still give significant load savings based on how you’ve described your proxy system kicking in because the Venn Diagram of proxy zones would only ever have three intersecting and never four.

[@AurorJeffrey]

Put a final way, the square mesh pattern means every server neighbors 8 other servers, 4 face-to-face and 4 corner only. The hex mesh pattern means every server neighbors 6 other servers, all as face-to-face. In square mesh pattern, a proxy can need replication across four servers. In hex, a proxy at most needs replication over three servers

[@AurorJeffrey]

Worth finally noting that regular hexagons are the shape with the highest number of edges that can tile a plane, so there’s nothing past this suggestion to worry about. Hexagons are best.

[@sinsaru]

I might get flak for this, but it’s the honest truth. As a technically minded person, I haven’t bought into any of the ashes of creation “hype” until this video. This sold me. The sheer amount of innovation this is bringing to the table has made me a believer in this project. More of these please!

[@Kyizwa]

I'm really happy to see how much everyone wants to improve the game to be at its best. Great job hiring these invested professionals and awesome job done by them.

[@CatSnievans]

Great presentation! I think I understood everything, haha

[@AshesofCreation]

Yay!!!

[@kevinodonnell6298]

Great update. Very interesting to dive into the tech behind the game

[@R4V1OL1]

My first thought at the start of the video was 'what if everyone gathers in one spot' 🤣 Everything is very well explained! Great work 🙌

[@TygerBlueEyes]

It's good to be an informed player. This video was great info.

[@AshesofCreation]

💕

[@WoodOx]

Tech streams (and art >.> ) are my favorite!

[@koolkoolkoopa]

I'm a CS student, on my free time I have this little hobby game project I'm working on and it's a MMORPG, of course, it's a very simple one but still, having to build the dedicated server and learning all about things like rollbacks, sending deltas, serializing, load distribution and how everything connects is super insteresting. So I LOVE having in depth reviews of the networking, there is very very little content since this is a very complex topic but for those inside of it it's amazing to look at.

[@ozzymandingo]

This was great, as someone that has worked in similar technology spaces (albeit not in game development) this was very cool to see

[@KingKascho]

I really liked this tech showcase its intressteding to see what U r doing behinde the curtains and getting Infos on stuff a Player dosn't See in His normal gameplay

[@aleksejszaikovskis3756]

This is so exciting to know more about the in-depth processes, so don't say it's not interesting. It's awesome! Thank you guys! Waiting this game so much!

[@wildterk6631]

This is massive! Great job guys, hoping for a timely release now with minimal bugs!

[@YourOnlyONEofcl]

WOW. That's crazy! I love the new techs you've created. We can finally get back “massive” in ”MMORPG”. I can't wait for this! Thank you for explaining it, I love it!

[@JakeyEX]

Absolutely fantastic update! Major props to the team for executing such a well done presentation, as well as making some really heavy technical info palatable for the core audience. Round of applause, excited to hear more about the bard and A2 date at the end of the month 👏🏼

[@melonadegg]

As the world biggest nerd and robotics engineer by trade, I absolutely loved this stream! Would be nice to see smaller tech heavy segments in other regular streams for the big nerds out there! Great job Intrepid team and well done on your breakthrough! :)

[@icloas]

Ill be honest, i am a developer for internal systems for businesses (automation etc.) but listening to this type of problem to solve... i think im in the wrong field. This sounds fascinating to sink your teeth in. I dont think i live close enough to work but just the sheer passion is heard!

[@daoshen]

Wow, that slide deck is art/science on its own! Love so much the team is sharing this work. Thank you all! 💖

[@Hat_Uncle]

I'm a complete networking nerd, this was awesome. I run a 10gb home network that's turned into a chore with the new SSH vulnerabilities. LOL I'm just biding my time now. Now it's Ramp Time for "Bardic Lore" and "Stand-Alone Character Creator"!!! I may or may not get into spot testing, but I didn't get to test as much as I would have liked in A1, so it would be no surprise if I didn't make the cut before A2 comes anyhow.

[@AshesofCreation]

💕

[@dylandaydream]

I can't wait to play this game love you guys being so open about development.

[@NedeljskiH]

We need more tech stuff like this. This was gorgeous.

[@kojiosita]

Amazing work! I can't wait to experience the final product!

[@Naruir]

You guys are doing a stellar job! Making magic happen! I'm super excited to see how this will work in game and the effects it will have for players. In cities around main npcs and shops is going to be the main test though, you definitely want that big social side of things.

[@zalmez7207]

I loved this, I haven't been following these updates very much - but I might considering following a lot more if it we get techy updates like this

[@ragnarlothbrockk9732]

watched it twice. That's how good this showcase was. Thanks intrepide! GREAT WORK!

[@gamershow5944]

Keep up the hard work us players cant wait to make it home. It will be super exciting to see the largest battles and fun game play ever created.

[@Sethz_aut]

Having played Ark and ASA I know that multithreaded replication is HUGE!!! Also very nice job on that slides!

[@Vulper]

I remember some of the general ambitious ideas how to solve UE limitations mentioned on one of the very first videos and I thought you just abandoned this as it turned out to be too difficult, but man, here we are. I'm impressed.

[@brandonanspach2801]

Thank you guys so much for your work. The work you guys are doing is what the gaming world NEEDS to move forward. I cant wait to help out in any way I can. Even if it is just on the forums. I also have a friend that will likely apply so I will send them a message today! also personally I LOVED the more in depth tech side of this. This is literally what I spend my freetime reading up on and tinkering with and its how I made my career what it is. This is literally the best video I have seen for this kind of tech and explaining how it all works!

[@aspirex8226]

Hey! Great choice with microservice architecture for your MMO RPG project. It’s an ideal fit! As a developer who loves optimizing service performance, I can really appreciate the noticeable impact of your work. Although my contributions are usually less visible, I can see the impressive results of your efforts. I’ve always wanted to work with gRPC, and it seems like you’re using it for cross-server RPC calls. Your implementation of load balancing, server scaling and proxying is remarkable-you’ve practically built k8s for gaming. Kudos to you guys!

[@sarkee87]

I don't understand network coding but because of my job I like to make things more efficient as an applied mathematician. I really liked the tech behind the scenes. I have seen similar attempts at this in other MMOs, most of which were abject failures. I have a feeling as a layman that things will be different here, you are going in the right direction, this will be huge step. Glad, I can support this project as a backer.

[@kayak118]

this was an awesome video. honestly its videos like this that get people excited about the game. hearing everyone's enthusiasm about what they do, and what challenges they have faced on making this game is so awesome. cant wait to see what comes next, and hopefully get a chance to play it. :D

[@AquaIicy]

Amazing video guys. I've been tuning in here and there for Ashes, but this showcase of how you're pushing the networking boundaries for MMOs is inspirational. Definitely makes me curious in providing feedback or even applying

[@BrutalGeek92]

I loved this stream! You guys are doing awesome work and I am in awe of the dedication and passion you guys are pouring into this game.

[@millertime8692]

awesome, great job! super excited about all of this!

[@stephenjones8105]

This was an amazing stream. Best one by far in my opinion. This really helps to appreciate the hard work going into this game and the future is off the charts bright. Thank you for this.

[@rerollator]

so excited to play this game in 10 years when its finally out

[@qwertpoo1]

As a nerd I approve this video thanks for the transparency ashes team you guys are changing the way videogames are developed.

[@AshesofCreation]

💖

[@roguepithlit]

Great overview of the server meshing. The effort you are taking to ensure stability of the game is exciting. I have one thought though. At 29:35 your slide says that a server is responsible for one grid component. If memory serves this is how EVE did their system, and in massive battles load was still an issue. Assuming you do not dynamically subdivide as load increases, it may be wise to see if you can scale the containers your dedicated server system runs on and allow for multiple servers/processes for a grid. Keeping those in sync via gRPC or similar technologies wouldn't be that difficult in comparison to what you have already described. With a true no layer system in place you may have thousands, or tens of thousands, of players in an area battling(if history teaches us anything it is that players don't really think about server load when they plan massive PvP battles). I LOVE that y'all are talking about how you plan on doing your backend!

[@markosz22]

It was freaking amazing and inspiring! You should definitely bring this to GDC as a presentation, maybe it would attract some people to the team. Also great to hear Alpha 2 purchase coming back, I missed my chance last year but I'll be sure to get it now.

[@smileychess]

To be honest, I'm always more interested in the underlying technology than the game itself. So I really appreciate the deep dive.

[@tokyodrifttt]

I didn’t understand everything but it was really interesting to listen to. Hopefully there will be KZbinrs that will break down on what exactly was achieved that wasn’t before and how this translates to players.

[@RotundaRaces]

I’m super excited for this game, as a comp sci guy from the era of c++, make sure your deconstructor methods are clean so you avoid memory leaks when players jump from server to server 1️⃣ idea would be to have transmitters to turn proxy copies to auth copies, and vice versa and know which properties need added or sent to server cleanup, very cool tech

[@gc-xv8bl]

The complexity of the tech used in mmos is probably one of the reasons why there's so few good mmos. Good job explaining this stuff.

[@dexagalapagos]

This was an awesome video. Best insight video I've watched on any game dev so far.

[@lawrencebob4250]

Awesome, Thanks for the in depth Update

[@ggGoldgg]

1️⃣ thing to know is I love this technical deep dive

[@dlaw555]

very cool breakdown. I can't wait for this game to come out.

[@helder1340]

For those unfamiliar with software development, here's some insides: Story points - unit of effort measure for a certain development task Backlog - it is the bag of all development related tasks that are yet to be done Debug - it is the way of checking the code as it runs, or register (log) all what is happening Server meshing - it is the mechanism of server interoperability which is the way they speak with each other to create a working server cluster Milestone - it is a set of things that once done fulfill the milestone and normally creates a patch or a minor/major version to be tested out or deployed Concurrence - entities competing each other for resources such as players asking for server responses Expensive in development - how much computing and hardware resources a certain thing consumes

[@MrSpetzz]

Can't wait till the start of Alpha 2 Might start making content over this and take a deeeeeeeep dive into the mechanics and seeing what is actually good and could use a tweak It has me hyped watching this game for the past 5 years slowly being crafted and made by people who actually care about the creation they are making. 1

[@RPG_ash]

Loved this update much more than usual. They are always great, but the tech is more interesting. I'd rather see this stuff than the usual "gameplay" stuff that what I think would spoil it for me when I do start playing. I'd like to experience that when the time comes rather than knowing what's coming. Amazing work peeps 👍🏻👌🏻🎉😁🔥

[@Tomasuu1]

as a gamer and a indie game dev i love these. these things are exciting :))

[@jonyanderson]

My dream, a true massive multiplayer sandbox, great job!!!

[@DakPrescottlol]

I recall a game called Rift and it was all cloud based server side. One dev talked about seeing 1300 people in one zone, no issues. That was pretty awesome.

[@IdhalarAlBaiesh]

It was a great stream, really missed these more technical updates and I'd love to see many more like it. Can't wait to start testing and see you all ingame! edit: Also, as a braver of worlds backer I'm perfectly fine with you selling more alpha keys. If it can help you stress test the server go for it, the more the merrier and more cash for you to work without pressure.

[@Bannonn_Digital]

Thailand people are waiting to play ❤

[@Rlifshitz19]

This sounds cool. Also sounds like an easy way to shoot your self in the foot though. The trade off for compartmentalization or decentralization is always complexity, and complexity is also difficult to work with. I am praying for this to work cause it sounds great.

[@ralaxgaming]

Just to give you some idea of public interest in server meshing discussion, I’m not an AoC backer or player and I probably won’t ever be but I’m here because of your work in server meshing. AoC and Star Citizen are the only games with static meshing AND working on dynamic meshing and there is a segment of gamers like me who have played games for a long time under the limitations of traditional MMO server architecture who are excited about any games being developed that are attempting to push past the limitations. That should give you some idea of the appeal of tech heavy discussions like this even outside your community. We know what you’re doing will seep out into other games in the years to come so we’re all rooting for you!

[@LilCassiopeia]

Was a nice live stream! We watched it live from Germany we went like oooh and ooooh and wow already 2 hours in was it supposed to be this long? :D

[@AshesofCreation]

How's the weather in Germany? We love a good oohing and ahhing. ;)

[@ParatrooperNinja]

As a physically impaired gamer, I so hope that you are able to implement controller support down the line! This game looks like it’s going to be absolutely amazing!

[@nikroth]

This is truly magnificent !! Thank you !!!

[@AshesofCreation]

🥰💕

[@d12duke]

Very complex to complete. Well done.

[@AshesofCreation]

Thanks! 🧠💪🏽

[@StefanHayden]

as a developer I love this stuff.

[@JohnAPrescott]

This stream was FIYA!!!!!!!

[@T3FAT]

They are so confident that they allow people to buy packs again! That is wild!

[@AshesofCreation]

WOOT!

[@Hat_Uncle]

well, actually, only the Beta holder can Upgrade Packages. The other people can buy A2 Only keys for a reasonable price. They have to walk a fine line there, make sure the Package holders feel fine with it, keep the price as low as they can, but make it feel like a Value for new people. I figure, based on my A1 package, value and components, that $10-$20 will be reasonable.

[@robertwiesner6825]

We as gamers absolutely love nerdy stuff. Look at Minecraft, Mount and Blade, Starcraft2 anything by Bethesda etc., modding is a huge part of those games. And there even are games that had been dropped by their original devs and are being kept alive by players like Dawn of War or LOTR BFME. Do not underestimate the power and nerdiness of the internet hivemind.

[@helder1340]

Game changer congratz IS! I hope combat Will be as well thought

[@israelnotario7413]

Gamers don't die, we respawn!. Looking forward to this awesome game

[@bitbraindev]

Software Engineer here. This is mindblowing what you've built there. Especially the dynamic gridding is something I hoped many other games would successfully implement. Dynamic gridding will become especially important once you have a large raid onto a caravan that moves across the world. Over time, the grid must adjust as the hundreds of players slowly move through the regions.

[@resident355]

The scale you guys are working in is a breath of fresh air, this model you presented is where tech is at today, readily available for anyone to try and implement. Glad you guys upped the game and sunk nights into this long term problem that gaming in general has been facing since forever. If this works, we salute you.

[@Nicolairoed]

Made me more interested in the game, thanks for sharing the tech (i have no idea what you're talking about ngl)

[@papadross]

I hate that I missed getting a copy of the game for testing. Really looking forward to playing one day.

[@weirdafters4007]

no one is playing at the moment so you didn't miss anything lol

[@ImBeyondTheBrush]

Love this stuff networking is my passion, AOC devs if you need more servers hit me up also based in San Diego and I can get ya discounted servers pc's and parts also sounds like a good excuse to visit the studio. 😊

[@jassimsalam]

The best development update I've seen ❤

[@nordiccowboy3271]

We've evolved into Ashes of Citizen!

[@TheYashakami]

I love this. Great video.