I REGRET Not Telling Dave Farley THIS about Approval Testing

[@lldadb664]

I was listening to the original conversation just yesterday and was thinking during that particular exchange that it might be a great opportunity for you two to pair/ensemble on an example to provide more clarity. I’d love to see that, but this definitely helped provide me with more insight. Thanks!

[@EmilyBache-tech-coach]

That's a good idea. Glad you liked the video.

[@craigjbass]

This is great. It never occurred to me to approach approval tests this way.

[@EmilyBache-tech-coach]

I'm encouraged you think so.

[@marco.garofalo]

I found your videos very well explained and well thought, I appreciated your take on using approval testings as a TDD/BDD tool, that said it didn't seem to be very practical. The main problem, for me at least, is that approval testing forces you to jump in and out the tests to understand input and outputs/expectations, disrupting the TDD/BDD cycles and adding a bit of a slowdown, whereas a more traditional TDD/BDD approach would just have those co-located. Similarly, when I try to understand a new piece of code I go look into the test suite, which in a case where approval testing was used, would still give me a partial understanding/view. Beside that I really enjoyed the exercise, also because challenging our assumptions is what makes us learn, so thanks for sharing your experience and expertise with us.

[@EmilyBache-tech-coach]

Thankyou for your thoughtful response, I'm happy you're willing to have your assumptions challenged! A couple of things occur to me - I don't find this 'jumping in and out of the tests' to be a real problem - I can understand a test failure usually just from the diff, often don't need to read the code. Secondly, the BDD cycle with a tool like Cucumber has a similar 'jump' between the feature file and the step definitions.

[@TheFoxesTale]

What an amazing example, this opens up a whole new world of possibilities for my complex integrated system testing. Subscribed right away!

[@EmilyBache-tech-coach]

Yes, sounds relevant for your situation. I hope it goes well!

[@danielirvine]

I enjoyed watching your walkthrough of this technique - it was illuminating! I often mix up specific names of testing practices because my brain just doesn't work like that - I tend to just do whatever I think will help in that particular moment without necessarily thinking about what is that I'm doing! E.g. the difference between TDD and BDD was never a huge deal for me, but I get that for many people it's important to differentiate between the two. I've also grappled with the problem of self-referential tests, but I never stopped to think about a process that avoided that problem. Now I know that there's a way :) Thanks!

[@EmilyBache-tech-coach]

Thankyou! I do find a demo does a lot to clear things up, especially when terms like TDD and BDD may not be terribly meaningful to people.

[@jangohemmes352]

I was lucky enough to win the privilege to host a Patreon Q&A session with Dave, and this video came up. I told Dave about the video and tried to play the role from your side of the argument. That was tricky, as I myself am not quite convinced yet. The main thing in this video specifically is that I think the example is just too simply converted into regular asserts inside regular tests. The same kind of sketching and modelling process of expected output can still be done, and to my mind it's better than leaving the IDE to tinker in text files. What I hope to have gotten across to Dave though, is that you really do model expected output before running the test, and are not simply accepting whatever the code produced. (In his words: "testing the code that you wrote is the code that you wrote")

[@EmilyBache-tech-coach]

That's really cool, thankyou for being willing to play the role from my side of the argument and talk to Dave about this topic. If he's started to understand that these tests are not 'self referential' then that's a big win! Your other argument here is that you could just use regular asserts and not leave the IDE and that this is a better workflow. I disagree, but perhaps that's a discussion for another day :-)

[@JhanaratoBhikkhu]

As an exercise, I used an approval test with the Leaderboard Kata in Python. I pretty much did what you did in this video, in my case implementing __str__ on the Leaderboard class. The main issue I had was the lack of integration with PyCharm. I had to manually approve files. Not a deal breaker though.

[@EmilyBache-tech-coach]

That sounds like a good approach! I agree better tool support would help.

[@banatibor83]

Hi Emily, I enjoyed the engineering room interview very much! I think you both are talking about the same thing under different names. I commented this on Dave's video but you both must have missed it :) Approval testing and Acceptance testing are both done in a BDD way, but done at different points of the software lifecycle. Approval test = safety net for refactoring Acceptance test = high level driving test for new functionality The tricky part is to realize when approval tests are used for adding new functionality they are called acceptance test. Hope this helps to smooth out the differences between you and Dave :)

[@EmilyBache-tech-coach]

I'm glad you enjoyed listening to Dave and my conversation in the Engineering room. The terminology around this is a little bit involved, and it's good to point out the different points in the software lifecycle where different words make more or less sense. However I'm not sure I fully agree with your analysis here :-) I hope we'll continue a constructive dialog - with Dave and you and others.

[@larserikaa]

I've certainly suffered from approving too large XML, JSON or manual prints. No way you can approve them safely when too big. But your usage of printers are just super exemplary. And I thoroughly love the story abstraction. If Dave doesn't change his mind now, I don't know who can make him. 😁👌

[@EmilyBache-tech-coach]

Happy to hear you recognize both the problem and the solutions I showed.

[@skipodap1]

Great Video!

[@EmilyBache-tech-coach]

Thanks!

[@matheusaugustodasilvasanto3171]

Where would you say these kinds of tests fall into the test quadrants of "guide development vs. critique product" and "technology facing vs. business facing"? The approved output files might be a bit too technical for business, but perhaps with different formatting they could rival FIT and Gherkin in readability. Where would you say these tests fall into the test pyramid? I think they're above unit-level tests, since they can test entire sequences of actions and can be used to validate any and all three fundamental assertions of tests: 1) results (by registering the return values of calls), 2) state changes (via direct queries before and after actions), and 3) external messages (by subscribing to a message queue). I don't think they should be the prevalent form of testing, since they involve I/O and that can slow things down once you've got hundreds or thousands of these tests. (Although admittedly, using a printer to output only relevant information allows the test framework to do a raw comparison between the observed output and the golden file, speeding the approval tests dramatically; most slowdown in characterization test happens due to large output files, or hacky regex expressions to deal with non-deterministic output.) I think they may be applicable to integration-level tests, specially when checking services get initialized/shutdown in an expected order. There may be some challenge in applying this technique to distributed systems (where events do not happen in a fixed order; property-based testing might be better for these situations), or very opaque ("non-testable") legacy systems, but there's definitely potential.

[@EmilyBache-tech-coach]

I think approval tests are on the 'guide development' side of the quadrants and can be tech or business facing. With the pyramid I think they are not strictly unit tests but could play that role if they are fast enough. They could be used at any of the other levels too.

[@BementalSea]

An observation/question about your approach to TDD. At around minute 10:00, with all tests passing, you say "the next step is to write some production code" which you proceed to do. And that makes a test fail. So we have: tests green->add new production code->tests fail. Doesn't this violate Uncle Bob's first rule of TDD? "Write production code only to pass a failing test". It should be: tests red->add new production code, tests pass. You repeated this pattern several times. My TDD spidey sense was tingling! This raises a question for me: does approval testing itself make it difficult to do TDD? Or was this backward approach just an accident?

[@JohnWilson-xl3rl]

This is a good observation and actually this process is much more similar to a TCR cycle than TDD but instead of auto reverting you are making an active choice to accept.

[@EmilyBache-tech-coach]

I agree this is an interesting difference in approach to TDD. I think the essence of TDD is breaking work into small pieces and very frequently getting feedback on whether your implementation currently matches your expectations. So no, this green-red sequence wasn't an accident. Approval tests tell you what your code currently does compared with what you previously approved, and allows you to easily update your test's expectations. I'm interested in @johnwilson-xl3r's observation that it's more similar to TCR. I hadn't thought about it like that.

[@craigjbass]

@@EmilyBache-tech-coach what you demonstrate feels disciplined enough to be equivalent to TDD but not exactly TDD by the letter, to me at least. I don’t think TCR always achieves equivalence with TDD, but this approach with approval testing seems to. I need to try it myself to fully form my opinions. With tcr I often dislike the lack of seeing the tests fail for the right reason before I start implementing, but when I use it for green to green refactors it’s amazingly valuable.

[@JohnWilson-xl3rl]

@@craigjbass with TCR you can "see red" and check that you are asserting new behaviour (rather than copy pasta and existing test) by changing the test to be failing once you have successfully committed. This will revert the code but there is nothing to revert, ie no loss of progress but you can be sure the new test was the behaviour you just coded.

[@JohnWilson-xl3rl]

Furthermore this is exactly what we are doing with Approval Testing, but as we are much more on the "outside" than TDD or TCR what we essentially have is this very elegant way of making incremental automated "acceptance" tests. You see this in the video as each approved approval becomes more like Emily's sketch (manual acceptance test). It really is a superb practice for the outside loop in double loop TDD. Thank you @EmilyBache-tech-coach for this video and the channel.

[@jfmengels]

Thank you for the video! When/why would you choose to use approval testing instead of "more regular" tests in new code? For instance, these tests you have shown could have been equality assertions against strings, such as `assert.equal(print(actual), "VendingMachine Display ...")` (let's imagine that the language you're working with supports multi-line strings that makes them pretty easy to view, just like you view your files). Is it mostly to speed up the time to update the tests with new results, as tools don't tend to modify your test code? Having the results in a file makes it harder (and more error-prone) to find the the result of the test, which makes it less usable as documentation, no?

[@EmilyBache-tech-coach]

Keeping the approved text in a separate file from the sourcecode has both advantages and disadvantages, as you point out. If the text is long, then it makes your test less readable to keep it mixed up in the sourcecode. This would also impact its value as documentation. I prefer to create tests where I can understand what the test does by looking at the approved file, without needing to look at the actual test code very much.

[@UserDirk]

@@EmilyBache-tech-coachI'm still struggling to get a sense of when to apply the Approval Tests. You mention "I prefer to create tests where I can understand what the test does by looking at the approved file, without needing to look at the actual test code very much.". But this somehow does not resonate with me. Could you elaborate on this perhaps?

[@EmilyBache-tech-coach]

@@UserDirk This is a great question and I don't think I can do it justice in a short comment here. I will make another video and try to address this better 🙂 Let me just say that I prefer Approval tests for larger-scale tests for bigger pieces of functionality, that need to check more than one thing. The approved file shows you all the important things that happened.

[@UserDirk]

@@EmilyBache-tech-coach Thanks, looking forward to it!

[@QmunkE]

Although you clearly can use approval testing with code like this as demonstrated, I think really the core of the argument is more whether it's worth doing in preference to (or in addition to) an xUnit style test. In cases where you already have good "traditional" unit tests in place, this kind of approach feels like it could add significant maintenance overhead to your tests and potentially couples unrelated tests to your implementation. On the flip side, I can also see that this allows you to spot cases where otherwise your test coverage might no longer be sufficient - adding a returns tray and verifying a coin doesn't get added to both the balance and the return is a useful test case you might not otherwise have!

[@TheLucferreira]

I dont know if I got it wrong but I thought that by "legacy code" she meant "code that exists but that is not tested". In this case, often the code is also very difficult to test with "traditional" unit tests, and then approval testing can help a lot. In scenarios outside of that definition of legacy code, there is still parts that are difficult to test using xUnit style tests, like parts that involves what is displayed to the user, and I believe approval testing can also help in this case. Of course, other strategies do exist.

[@EmilyBache-tech-coach]

If your code already has good xUnit style "traditional" unit tests then you are in a good position to begin with. Approval testing can help when you want to test bigger pieces, like a whole microservice or subsystem, and you would otherwise want to assert several things. Approval testing is also useful for code that doesn't have tests - legacy code - and that's where Dave and I do agree.

[@pierre-antoineguillaume98]

I like what you describe, in the end the printed results are less cluttered than their code counterparts. Good luck for your channel, I have shared it with my lead coworkers ! Thanks your your work, we use the kata of your repository in php !

[@EmilyBache-tech-coach]

I'm happy to hear that! I hope your lead coworkers find it useful.

[@PeterGfader]

13:15 Cool to see this helper tool in action. Maybe something that IntelliJ/VS will incorporate in the future. PS: I hope the teams behind these tools would focus more on "testing features" and less on "debugging features" :/

[@EmilyBache-tech-coach]

Yes, I think there is a lot of scope for the toolmakers to better support approval testing, and developer testing in general. You can do quite a bit with plugins too - I think this analyze script could be a plugin in future, and that is quite achievable. Just need to find the time...

[@PeterGfader]

Very cool. THANKS! @@EmilyBache-tech-coach

[@haradan8887]

So this is just a different way of making assertions you can use for any automated tests? It seems like there's a potential disadvantage in needing to re-approve old tests. Say you add a new field to an object you're asserting on. If you assert on object equality, you can update your assertions implicitly by setting a default value for a new field. If you use approval testing, you need to either re-approve all your tests with the new field, or set up your printer to avoid the old tests printing that field at all, or avoid printing it if it has the default value. Do you think it's an extra effort of design to get that right? Do you default to one way or the other?

[@EmilyBache-tech-coach]

Yes Approval testing is an alternative way to do assertions. Re-approving old tests is not a particularly big issue if you have many that have changed in the same way - with tool support it goes very quickly. You are exchanging the complexity of writing assertions with the complexity of writing a printer. I don't think the amount of code needed is very different, but the failures are easier to understand in approval testing.

[@haradan8887]

@@EmilyBache-tech-coach Thank you so much. I'm so often frustrated by awkward output from an assertion failure. It sounds like this can solve a lot of problems.

[@bernhardkrickl5197]

I still don't get it. I get that with characterisation testing you just accept whatever the code does and nail it down by writing a test for it. Then I understand that with approval testing you think of what the code should do and write a test and then make the code do it. But what is the difference between that and regular TDD/BDD? Also, you didn't even precisely do that. You had a PNG of the result but the executed test didn't check against the PNG, instead you changed the implementation first to produce a red test and then adjusted the text files from your printer to make the test green, comparing it with your own eyes to the PNG. So, what in all this is the crucial part that makes it Approval Testing and sets it apart from other things?

[@EmilyBache-tech-coach]

It's similar to both characterization testing and TDD. I think you've done a good job actually of explaining the differences.

[@ewinslow822]

Another word I hear for this is "snapshot testing" because instead of making targeted assertions on the expected outcome, you are just printing a snapshot of the state of the system at some point and relying on humans to be able to judge whether the snapshot "looks right." IMO an improvement would be to instead break this into two non-snapshot tests: scenario: when i insert a penny - the penny ends up in the return drawer - nothing else about the system changes

[@EmilyBache-tech-coach]

Yes, some people use this language calling it a 'snapshot', however I don't really like to use that word. It implies a kind of casual impermanence. I'd rather talk about output that I explicitly chose to print and approve. Of course you could re-write this as two tests, as you suggest. I think the approval test version is easier to understand and less code overall though.

[@tongobong1]

Dave Farley is a guru that doesn't quite understand software testing and to be honest many gurus don't understand it.

[@EmilyBache-tech-coach]

I wouldn't go so far as to say Dave Farley doesn't understand software testing. I think he has judged Approval Testing without really understanding it though.