Where The $@*&% Are Your Tests?!

[@babgab]

My comments: 4:30 I've found that conventional wisdom isn't all that helpful here. If you verify only a single assumption per test (which I've heard called the "one assert rule") then you end up with loads of duplicate tests, because any given operation may have multiple assumptions or implications. The "one assert per test" rule just isn't practical; one *state change* per test is the one I follow instead. 7:30 This is a really good point - automated tests are a form of executable documentation! I don't think this point is made often enough, at least in places where gamedevs can hear it. 11:16 This is also a good point - but to my mind this is just more reason to make the tests as simple as possible and make it understood that if you're changing code spec on purpose, you're allowed to delete the tests and rewrite them. A lot of us are so averse to rewrites that treating unit tests as disposable is unpalatable and I think that's something we need to overcome to really adopt automated testing. 12:57 This kind of integration issue is a reason I've heard cited NOT to do automated tests, or at least not to do unit tests. There's a perception that "the real bugs we deal with" are mostly integration issues, which I think is not untrue for gameplay-level code but not all code needs to be that coupled; coupling can be deferred to a "glue layer" making the unit tests be signals that the individual units are doing what they're intended to do. 15:30 From talking to "testing gurus" myself, my understanding is that the need for test speed is specifically for test-driven development/"red green refactor" workflow where the test is run at least three times for every new assumption added to the system. The most important thing in speed is that they aren't so slow that nobody runs them, in my experience. I've experienced test suites that took *four hours* to run. That is far too long. Nobody ran those tests if they could help it. 16:45 The reason to run in memory is NOT just about speed - it's about isolating the test from other components of the machine that can fail. If you can avoid it, you don't want your tests to fail because the test machine in the CI farm ran out of disk space, or its GPU rebooted and the test wasn't a graphics test, or because a network share went down, or any number of similar issues. It's not about speed, it's about reliability. Devs lose confidence in unreliable tests and stop running them, so the tests become pointless and fall out of sync with the code. I have watched this happen personally. Do NOT involve disk if you can avoid it. Sometimes you can't, but then one ought to be explicit about this. This is also a problem with large granularity tests; did my test that has a car drive at a wall and then avoid it fail (for instance) because my steeering is broken? Or because the car's engine is broken? 26:23 Lack of buy in is THE problem. If you're not motivated to write good tests then you'll write shitty ones. If you're not motivated to write tests at all and are only satisfying a management directive, you'll write shitty tests. An appetite for testing has to be sparked at the grassroots level to succeed. Programmers are resistant to ideas they can't see any justification for and management is not good at justifying tests. Test guru consultants aren't, either, because they don't understand the constraints of the game industry. It has to come from us and we need to lead by example.

[@evannibbe9375]

Godot “testing” is called running the game and seeing if it crashes, then trying all the inputs you set up and seeing if it crashes. Then, once the game is built, you check if it runs on the target systems.

[@gJonii]

Also setting up tests that use hard drive is a pain because now you need to add logic of maintaining, initializing and checking state of that drive, with all the possible things that can go wrong there... As part of a test, which should be as easy to understand as possible. You could write whole programs for that purpose with large test suites to their name, and you're now cramming it as initialization to some single test? Good luck.

[@peculiarnewbie]

can you apply for a talk please good sir/mam

[@babgab]

@@evannibbe9375 Which is slow, tedious, and error-prone. Automation is faster and more repeatable. Human testing is good at uncovering corner cases that nobody thought of, which is why we can't just replace QA with AI, but automation is better for the rote stuff that anybody could've thought of.

[@olon1993]

I want to explore this space so much more. As a software engineer coming from the business sector I wonder how these giant games get by without rigorous unit test suits. QA and developers are in a position where they have to brute force the problem, but we all know there has to be a better solution. Great talk! Wish you had 10x the amount of time to dive into this topic in detail.

[@SunSailor]

For our current project, we go the full way for automated testing and to this point, it was a gain for sure. We use data binding a lot, which goes very well hand in hand with testring and refactoring our systems is much safer now with these two approaches. A lot of systems are extremly independent now and this makes live simply easier. True, as we create a RPG, the complexity of the game defining systems is much bigger than in, let's say, a shooter, but I would even do such with a wide test coverage as well. Great talk, would love to hear more from you, Kevin!

[@JM-dq7xn]

At work (non gamedev) I have very strict testing policy - every change needs a test. And it is often very straightforward to do with back-end software. But in gamedev... like *how* do you write a test that tests if, for example, a character jumps? Especially when that scenario involves input, physics, animation, etc...

[@Bobbyrewdas]

You make a game scene in setup, spawn a player, call Jump, wait N frames, and check if their y value has increased by the appropriate amount, whether their AnimationClipInfo clip (to use a Unity example) is correct, etc.

[@pik910]

IMHO writing automated tests for jumping would be overdoing it and the optimal way to test it is playtesting. The sample data would be hard to produce (considering edge cases), and would not tell you if it looks and feels right, which is the actual requirement. Input, physics etc. should be tested independently, that is the engine writers responsibility. Jumping is a relatively trivial combination of those systems. What I think is good practice is to have a test scene for player movement, where you can playtest a bunch of edge cases in quick succession. There you could also check more mathematical requirements, e.g. that max jump height is never exceeded. Then you have a reasonable degree of confidence that player movement works and might learn what kind of limitations level design deals with.

[@theobrominator]

As a web dev/eng by trade and starting indie game development, I feel that game engines and frameworks are really behind in test tooling. Most web focused languages and frameworks have at least one recommended unit test framework and usually at least a few 3rd party ones exist - golang has one built in. All with good documentation. Unity (UTF) and Unreal (UUnitTest) exist but the documentation is pretty sparse and in Epic’s case mostly statically generated. Even in Godot - the unit testing docs refer to plugin development. All of this is to say that there needs to be a lot more investment from the tool(chain) authors to reduce the friction from the start or adoption will always suffer.

[@fxcallowkids4344]

I think game is all about feel, no matter how good the structure and integration is, if it doesn't feel satisfying then it's pointless, implementing and checking quickly is the right and financially responsible thing. Creating games is not like making applications where everything can be planned and measured precisely, so unit tests are in an ambiguous position and can even be a waste of time. because of that, there aren't many people asking for that feature, so features like this are not well maintained.

[@pik910]

@@fxcallowkids4344 The idea is that testing produces higher quality code. Which means less bugs that are more easily fixed. Also you get to know when you break something in complex, interacting systems right away. This results in better maintainability (which can make or break a project) and is ultimately faster than winging it, because of how much time (and nerves) bug hunting takes. In reality a lot of stuff in games requires play testing. In areas more suited for testing I think it is a tradeoff between the work required to implement testing and how likely you are to produce bugs (how complex the system is, how much the code is supposed to be reused etc.). People probably do not test enough overall, making their lives unnecessarily harder.

[@fxcallowkids4344]

@@pik910 in theory it's very useful, but in reality, games are always about iteration, people can suddenly change the whole system they've built if it doesn't feel like it won't work, you can search for examples of triple A games that change its genre In the midst of development, it's not a secret, and it's only a small part of the obstacles and challenges in developing game, I'm not here to discourage or convince you, just sharing my views, unit tests are great, and great if you want to use them, but also consider that investors don't care how pretty your code is, what they care about is, when you deliver your product, that's why you sometime can easily find bugs in game, because the truth is we don't have enough time to create high-quality code.

[@theobrominator]

@@fxcallowkids4344 I learned the wrong way a few times how important tests are in the long run 😂. I don’t really fault devs though. Most of the time there’s a tight deadline set and if something not crucial passes a difficulty threshold it gets deferred or forgotten a la tech debt backlogs.

[@pik910]

@@fxcallowkids4344 Your last sentence is what pro-testing people disagree with and that is the main point I wanted to make, because I consider it to be a profound insight. Low quality code is at best faster to produce in the short run but the bigger and complexer things get it comes back to bite so hard it can total projects. Here is a video that makes this point for anyone interested: kzbin.info/www/bejne/kJ7Kk52Xh8ujlbM As for iteration, yes definitely. On a sidenote, I think when games make massive changes mid development in general that should not be aspired to, that should be done during prototyping. In the examples I am thinking of there was either mismanagement or politics (thief, re4) involved. That aside I would argue that when you have to break stuff, which is healthy and common, then code quality gets put to the test, especially when the code base is big. Then you want to have small modular chunks of systems that you can verify and change individually, because you cannot rewrite everything and you want to limit the amount of complexity you have to deal with. In that case it would be nice have unit tests that tell you which parts work and where you broke something. I've seen bugs caught by unit tests where some innocent looking change broke something in some specific OS. (Also not trying to convert anyone, just thinking loudly.)

[@pgasnow]

WHAT'S THE CENSORED WORD I'M GOING CRAZY OVER HERE

[@Guill0rtiz]

The world might never know...

[@filiplaubert5001]

Its just regex.

[@PropaneWP]

If this is an attempt to not get demonetized, I'm pretty sure the word is "Palestinians".

[@TheCrewExpendable]

@@filiplaubert5001 it’s the word you say when trying to write regex.

[@THEM0J0MAN]

its not Fuck?

[@BBdaCosta]

I want to listen to Kevin talking all day about that. Awesome tips and very well explained. Thanks

[@ArturCzajka]

17:12 theory goes, that you build your test first, and then you end up with the main code that is easy to test, and you avoid doing mockup of the entire environment ;) I'm amazed, that game dev industry managed to avoid automated testing for so long :P I started seeing presentations about testing on GDC since only around 2015 :O (but then I only view it on YT, maybe The Vault has more) BTW, see also The Talos Principle testing talk - it's great :D

[@captainnoyaux]

Which talk ? Didn't find it

[@ArturCzajka]

@@captainnoyaux "Fast Iteration Tools in the Production of the Talos Principle" - I see it's no longer on YT, but GDC Vault has an open-access copy

[@captainnoyaux]

@@ArturCzajka Thanks to your comment I found it, it's around 20minutes long. Thanks a lot I'll watch it

[@babgab]

Some gamedevs were doing automated testing as far back as 2009. Not to the same all-encompassing extent as other industries, bt I have a copy of "Game Coding Complete, 2e" on my desk and it mentions automation.

[@dthurn100]

I imagine that testing in the game industry is very genre-specific. Games with lots of complex business logic probably have more tests than games that are more about visceral game feel. I bet that the average turn-based strategy game or card game has a lot more tests than the average FPS or platformer, for example.

[@Bruiserjoe]

Both are equally hard and both require similar amount of time to perfect.

[@chris.dillon]

For something like game feel, you can't teach a computer what that means (yet?) so you don't do it. The same happens with UI. You just have a hot reloading and fast feedback loop and iterate on design. Then you extract out logic that is equally small and testable which you can teach a computer what is right. So, you cannot test everything and cannot test everything in the same way. That's why you'll find that automated pipelines (CI definitions in yaml) not only have many styles of tests but they increasing in "fanciness" as they go. But yeah, I think different apps have different resulting test characteristics (coverage %, style, speed, libraries, approach, how much overlap and duplication depending on what is considered risky or important). I think there are some universal truths across projects, or at least some hard won wisdom. Every project is different but I think a lot of projects think they are super special. "We don't have time for tests". Unless you are making a game for yourself (and even then), every game is tested eventually (by users). So the phrase lately has been about "shifting left". In other words, fast feedback loops. There are many aspects of gamedev I think that makes fast feedback tricky. AI is feeling this too but they are just starting out. I think, the the end, fast feedback is table stakes. So the tools are going to have to provide it. There's just too much value in it.

[@babgab]

It's actually the other way around - complex game logic is often *harder* to test, because generally complex logic needs to talk to a lot of things = more things need to fit in the test harness or be mocked/faked or the test needs to be done at a higher level of abstraction (=usually flakier, trickier to get right). That being said, FPSs are more complicated than you'd think, especially once networked gameplay gets involved. I'd expect that card games or TBSs are easier to test because the state space is more constrained than a sandbox FPS or anything real-time for that matter. It would be easier to change one thing in the game world and see what the effects are.

[@MidnighterClub]

Could you reference where you got that a unit test should test a "single assumption" about the unit of work? I do that myself, but I've got into arguments with other developers who like to use triple-nested loops and test everything about their method in one go, which I feel is counter productive. I must have read that or gotten taught to test only one thing at a time at some point, but I've totally lost whatever source that was.

[@snook5555]

I could be wrong but I believe it's just the best practice and how it is generally taught, and that you are right in that it's counter productive. As doing a bunch of tests inside a single test obfuscates the test code a little slowing debugging slightly (which adds up). Also by having separate well named tests you can see in the UI behind test runners a named item telling you exactly what test failed and can reduce having to dig through code to find the issue. There probably plenty of other reasons.

[@MidnighterClub]

@@snook5555 Thanks for the reply. This was what I understood too, and I was surprised to see someone do something so different. I'm going to be on the lookout for authoritative sources though since they might be handy if the situation arises again.

[@KyleLuce]

@@MidnighterClub not sure on origin in this video, but it's a point made in the text "Clean Code". Martin states something along the lines of. "each test should target only a Single Concept.". This is also a parallel with SOLID , in that the test function has a "single responsibility". I'm guessing the definition here is passed from these sources (or from some parent reference).

[@navneeths459]

Not a game dev, but doing automation testing for mobile app. Good to hear more on the topic.

[@andrewallbright]

I wish I had this guy to talk to IRL. I love testing. However; as a hobbyist game dev when I ask game dev online communities about testing it doesn't end well, oddly enough. I predict that testing is going to be a main stay of programming in the next decade and the big winners will be the people who spend time figuring out how to make effective use of tests. YMMV!

[@babgab]

Probably because there are a few prominent "influencers" who are strenuously opposed to it because they don't understand the use case, and the aspirants copy them because they thinks it gives them more credibility to parrot what some other indie dev who "made it" says

[@bluezamx814]

After a horrid experience working on a turn-based combat system which would benefit a LOT from unit-/ integration tests, I'm diving into this topic as well. Very informative video and a nice starting point!

[@MichaelMurkovic]

Good talk Kevin. Not a game dev but so funny how much overlap there is with testing & ci in enterprise software dev. Liked your suggestion to not mocking/being too atomic. Boosts that code cvg as well

[@enikey87]

You can't see unit testing without Kevin's Dill eyeees! (sign it along with Slipknot - Eyeless).

[@snook5555]

I love Unit tests, but I don't go for code coverage. As many tests don't have much value in testing, after all testing a function that just adds two numbers is a waste of time. Also unit tests is a better source of code documentation than either comments or word documents as they tell exactly how a method is intended to work and are more likely to be maintained (provided you setup a pipeline to run tests automatically when commits are made, and refuse commits with failing tests). Doing test driven development (TDD) is also great skill to learn, as forces devs who like to jump into "just writing code" to think more about the structure of their code without them really realising it, and since testable code is generally better architecture and results in less refactoring needed down the line. Also writing tests to replicate bugs is great, as then you fix the bug and have a test in place to prevent regression. In all though writing tests is a skill that takes quite some time to learn, most give up. But it does save time on refactoring if using TDD and more in general in finding bugs. It does cost time writing, but not by much once you get the hang of it, and the extra time is more than worth it. At least this has been my experience.

[@sillybuttons925]

Very much applied to data science as well.