Multi Threading Model in Paradox Games: Past, Present and Future - Mathieu Ropert - CppCon 2021

cppcon.org/
github.com/CppCon/CppCon2021
---
Paradox grand strategy games are all about simulating history through various lenses (politics, economy, warfare, diplomacy, demographics...) in real time. Over the last 20 years, our games have become more complex and more demanding on CPU computations.
In this talk we will see how the threading model of the game simulation evolved over time to try and deliver the necessary throughput and latency required to keep the games fluid with more and more complex systems.
We will focus on 3 models, the "old" which is shared by some long running production games such as Europa Universalis IV, Hearts of Iron IV and Stellaris, the "current" model brought by Crusader Kings 3 and finally discuss some thoughts and ideas of what future games may experiment with.
---
Mathieu Ropert
Tech Lead, Paradox Development Studio
French C++ expert working on (somewhat) historical video games. Decided to upgrade his compiler once and has been blogging about build systems ever since. Past speaker at CppCon, Meeting C++ and ACCU. Used to run the Paris C++ User Group. Currently lives in Sweden.
---
Videos Streamed & Edited by Digital Medium: online.digital-medium.co.uk
Register Now For CppCon 2022: cppcon.org/registration/

[@peregrin71]

Slides 60-61 Basically you need a dependency graph between the entities you want to calculate (e.g. C2 depends on C1 and U1), topologically sort them and then use a threadpool (with less threads then you have physical cores to allow the render thread to occupy one core). The next task to pickup is the firsts in the list that has all its prerequisites calculated.

[@RobBCactive]

Measurements show games don't utilise much of a recent core 6 wide core (IPC < 1), because of lots of branching and lack of tight loops. In one experiment turning off cores and having SMT actually consistently boosted frame rate in 2 different games. SMT can mitigate memory stalls and misses, while a thread falsely appears to be 100% when it's actually idling. So, having higher L3 per core and greater watts budget might explain such results.

[@dennisrkb]

Loved all your talks this year. Thx a lot for sharing!

[@RobBCactive]

Apache on Linux used a pre-spawned process model forming a pool of servers waiting on client requests, when just one would be woken up. In reality threads block on memory access, nevermind disk i/o so the 1 per CPU idea was not correct, plus sendfile(2) allowed a server to stay within the kernel space once the initial fancy request handling was done avoiding context switches (kernel MT on uni-processor). Perhaps the "toy" Apache port on Windows used 1 thread per request but Windows was 2nd fiddle with LAMP servers dominant.

[@DaveFeedBackGaming]

When speed 5 is too fast 😇

[@marijn211]

The increase in boost clock speed is still very significant, in addition to IPS improvements with the same frequency. AMD bulldozer, quite infamous for...not being great, had the main 3-cores that also had HT I believe.

[@pandaDotDragon]

So the solution is a fine grain dependency graph consumed by light processes executed on a handful of threads?

[@PedroOliveira-sl6nw]

Great talk!

[@h1tzzYT]

very interesting presentation! i Know strategy games are your field, but i would love to see similar presentation but on different genre of games as i imagine development for multithreading would be quite different.

[@lkuzmanov]

What I'm seeing during the demos reminds me of why I stopped playing Stellaris some time ago (and probably for the same reasons) - main-thread bottlenecked engine with an underfed graphics pipeline causing the game to update frames at a choppy 10-20 FPS regardless of the machine the user throws at it. To me this would still be just unpleasant to play. I imagine that the dev team is doing their best with the resources at their disposal (and I realize Paradox is neither Naughty Dog nor Bungie (thinking of their 2015 GDS talks on engine multi-threading)), but the result really isn't acceptable in 2022. I'm not writing this to insult the presenter, but in the hope that the signal will reach the responsible budget owners.

[@RobBCactive]

Well I was using the DX11 version years before that and even had higher FPS on potatoes with APUs. Since I chose to lock to vsync so the game ran at an even pace. The loading of files and such improved immensely 2020/21 and they continually have improved performance, it is however wise to select some options and have auto saves local not cloud.

[@guncolony]

I think the multithreading must be turning itself off when the AI spams divisions in hoi4 😆

[@petermuller608]

While the contents of this presentation were very engaging, the presentation was unfortunately very taxing. Try staying mute instead of filling with sounds like aaaaaaa

[@ABaumstumpf]

Multithreading - it is not hard to do work on many threads. it is hard to get the data from one thread to another efficiently. Thankfully at my company we are in the situation were we have many Independent sub-processes were the heavy lifters are mostly for communication and data conversion (1 process talking with X subsystem) so in general we only have multithreading for communication as that is the only place were it really is necessary and for all other processes a single thread is just fine - we run more processes than the system has threads available anyways. and it does not matter if data-exchange or manupulation takes 5 ms as we are not dealing with hard/soft-realtime, at most some user-interfaces (were a person has to do some physical stuff inbetween) is a tiny bit slower. And the good old - devs get worse machines than people that literally only open Word and Excel. So seriously - at work we get lowpower laptops, most people here have something like Coffelake/icelake-U processors (15W variants) while the software we work on is designed to run on large servers (>16core Xeons with >64GB of memory), and parts of the software can not even be started unless you got at least 128GB Ram... But still interesting to see how a game-engine deals with threading. In general splitting work by "system" is simpler and has less issues with syncrhonisation-overhead. User-input, AI, gamestate, physics, graphics - those are rough components that many games have that are mostly independent of each other.There is little reason to start the graphic-processing after the entire game-logic if the game is not really fast-pasted like a racing-game or First-person-shooter. In most games the user would have no clue if the graphics shown is from the state of the game 1 frame ago - specially for any turn-based or pause-based games. But if you have a single system with a very large pool of work to do it is often a good idea to not prematurely section to work prematurely into large chunks. So say your UI, user-input and Netcode are all done very fast but you just have massive numbers of units. So when updating units you do not want to chunk that by countries but have all the work in a single pool. And then spread the pool of similar-time tasks across whatever number of worker-threads you have. For thread-locatlity it is helpful if you provide the data in an ordered way.If you have 1 large chunk of logic that takes way longer than the other systems - say updating units: For example if you want to update 40 000 units across 20 countries across 5 threads (less threads than available) you would try to spread the units such that the units of 1 country mostly end up in the same thread when possible. A single thread constantly working with data from 1 country likely is more cache-friendly cause it does not need to reload country-specific data. And the number of threads used can easily be determined at runtime with simple heuristics like "The number of threads assigned is the number used for the last computation +- how much faster/slower this task was compared to all other tasks". Basically if in the last frame all other tasks finished in 100 time-units and this task took 200 units than add another thread, but if it only took 30 units but had 2 threads than reduce it to 1 thread now. (have had to use something similar to an event-system and worked decently at keeping all 88 cores saturated - 4x 22 core) Or if you have things that are Independent (say resource-production, unit movement, diplomatic relations) then those could be started at the same time. Thankfully most of my performance "improvements" had nothing to do with slow code but rather our system just having a bad tracing implementation and using HDDs instead of SSDs. So to "improve" performance it often was enough to reduce the lines of logging by combining lines into single longer lines of code. (Yes - enterprise-software were i got more than 50% increase in performance by writing out something like "items got finished" instead of writing each individually.

[@erick_ac]

I like that game!

[@e1123581321345589144]

EU4 for is my most played game on steam, but my favorite, by far, is Stellaris.

[@DaveFeedBackGaming]

Max😾

[@e1123581321345589144]

When Cities Skylines 2?