Prag Dave Talks Agile, Waterfall, TDD & MORE (Dave Thomas) | The Engineering Room Ep. 18

Dave Thomas joins Dave Farley in the "Engineering Room" to talk about agile vs waterfall, how software developers ought to look at software testing and gets deep into some of the interesting edges of programming, like algebraic effects, state and immutability and implementing monadic do blocks.
Dave Thomas a.k.a. Prag Dave, is one of the authors of the influential software engineering book 'The Pragmatic Programmer'. He's also one of the original authors and signatories of the agile manifesto, an experienced speaker and a thought leader within the software community.
-
⭐ PATREON:
Join the Continuous Delivery community and access extra perks & content!
JOIN HERE ➡️ bit.ly/ContinuousDeliveryPatreon
___________________________________________
🙏The Engineering Room series is SPONSORED BY EQUAL EXPERTS
Equal Experts is a product software development consultancy with a network of over 1,000 experienced technology consultants globally. They increase the pace of innovation by using modern software engineering practices that embrace Continuous Delivery, Security, and Operability from the outset ➡️ bit.ly/3ASy8n0
___________________________________________
📚 BOOKS:
📖 The Pragmatic Programmer: your journey to mastery, 20th Anniversary Edition ➡️ amzn.to/3EdXvBm
📖 Programming Ruby 1.9 & 2.0 : The Pragmatic Programmers' Guide, by Dave Thomas ➡️ amzn.to/3C1oUYq
📖 Dave’s NEW BOOK "Modern Software Engineering" is available as paperback, or kindle here ➡️ amzn.to/3DwdwT3
and NOW as an AUDIOBOOK available on iTunes, Amazon and Audible.
📖 The original, award-winning "Continuous Delivery" book by Dave Farley and Jez Humble ➡️ amzn.to/2WxRYmx
📖 "Continuous Delivery Pipelines" by Dave Farley
Paperback ➡️ amzn.to/3gIULlA
ebook version ➡️ leanpub.com/cd-pipelines
NOTE: If you click on one of the Amazon Affiliate links and buy the book, Continuous Delivery Ltd. will get a small fee for the recommendation with NO increase in cost to you.
-----------------------------------------------------------------------
🖇 LINKS:
🔗 The Agile Manifesto ➡️ agilemanifesto.org
🔗 Agile is Dead ➡️ pragdave.me/thoughts/active/2...
🔗 Algebraic Effects ➡️ overreacted.io/algebraic-effe...
___________________________________________
CHAPTERS:
00:00 Intro
01:06 Welcome Dave Thomas
01:39 “‘Developing with agility’, not ‘Agile Development’”
03:22 You can’t Buy a pound of agility
03:51 Agile is Dead
04:20 The Birth of the Agile Manifesto
05:48 The Impact of Commerce
06:52 Values Not Rules
09:17 The prevalence of Imposter Syndrome
10:45 We Have Got Better at SW
12:35 How do we rule out the bad ideas?
16:01 Avoiding Test Nazis
19:08 Does TDD teach better design?
21:57 When is a bad idea a bad idea?
23:18 The ‘what I did on my vacation’ software book
25:09 The Reason Waterfall doesn’t work
26:10 Small steps and experiments or up-front design?
32:18 What are the boundary conditions of your code?
34:55 Being thoughtful about the problems we are solving
36:33 The danger with TDD
38:24 It’s OK to do some thinking up front
41:10 Iterating from both ends
42:38 Building intuition
45:56 Expertise is about not having to think about something
48:19 How would you structure a Computer Science Programme?
51:20 The Ideal way to learn to programming
55:21 The duty to pass-on learning
57:40 Immutable State is a key tool
01:02:24 Immutable Code
01:07:53 Where does state live or “Never listen to what I say” 🤣
01:09:28 Algebraic Effects
01:16:00 Using “Monadic do Blocks”…
01:20:56 Separating the transformations from the reductions in your code

[@ContinuousDelivery]

Do you enjoy conversations like these, about different approaches to sw development? You might enjoy the CD Discord channel - available via our Patreon Page from £2 a month ➡ bit.ly/ContinuousDeliveryPatreon

[@ugandandev]

Massive knowledge dispensed here

[@alexandre8869]

It was really pleasant and very instructive to see these contradictory and complementary exchanges of ideas on such a fundamental subject as the underlying vision of a development methodology. Keeping a journal of ideas tested and to be tested is a good trick. This conversation between two gentlemen with mutual respect for each other's vision must be the reference in the profession and should be watched by apprentice engineers. I can testify that I also learn a lot by looking at the history of computing. Thank you so much Dave(s) for being involved in these knowledge transfers.

[@dougr550]

I think the conversation is Agile vs Project Management as waterfall is a symptom of PM. Agile is rooted in design thinking. Read the definition of design thinking and then try to explain how a schedule would make any sense. The problem is that most people, both technical and business managers can't figure out how to manage this. "Design thinking is a non-linear, iterative process that teams use to understand users, challenge assumptions, redefine problems and create innovative solutions to prototype and test. Involving five phases-Empathize, Define, Ideate, Prototype and Test-it is most useful to tackle problems that are ill-defined or unknown." Definition from Interactive Design Foundation

[@ssg6499]

Appreciate Dave F for hosting this - super important to have nuanced conversations like this; Probably his best interview imo. I didn't know much about Dave T before this interview (other than his name being on the manifesto), but I am shocked how much I agree with him: Will definitely check him out more.

[@peppix]

3 years ago I had the chance to manage a team that was developing mainframe software in a large organization that was pushing agile methodology in all the other departments. So I did ask the team to introduce some agility in their waterfally project. It was a big success and a great learning experience. There are 50 shades of agile indeed 😊

[@szeredaiakos]

I sooo agree with the apprenticeship idea. I had a couple of masters during my life and they where priceless.

[@theodorealenas3171]

Dave Thomas talks about state in such an elegant way, he reminds me of high level artists talking about gesture. Partly because I understand neither.

[@orrmadar]

Wow, that was a really good video. I really enjoyed hearing the different perspectives about testing, and I totally agreed with how we should teach programs, immutable states, and the dynamic event style of code. Thank you for your time, and keep doing your good work

[@logiciananimal]

"But Socrates, can virtue be taught?" I think the learning agility is exactly parallel (in many ways) to the paradox of learning in Plato's _Meno_. My ancient philosophy professor back in the day points out that the dialogue ends inconclusively, but with the twist that just maybe the intention is to show that virtue can be taught, but only by working through an example and living it. So in this case ...?

[@phoenix-walker]

I think i understand Dave *T's point of deleting the tests. There are many times, especially in the early phase of a NEW project, where the cost of writing and maintaining the test outweighs the benefits because you might end up rewritting 90% of the code. You can be really in a more exploratory phase. However, this is not sound advice for maintaining a large "legacy" system over many years. I cannot stress the amount of times a solidly written unit test has caught an error in my code as I wrote it.

[@manishm9478]

I wonder if for such projects it is better to transition to full acceptance level tests (given then when style)? Only thing is they are expensive to run (take a long time) so the developer doesn't get immediate feedback like they should with unit tests

[@jrgalyen]

Yes! Experiment with practices! This is the “and-also” mindset. For example, we can support trunk based development “and-also” feature branches w/ script that keeps feature branch up to date. Stop considering ideas dumb. Be scientific minded. Well said, Dave Thomas!

[@jrgalyen]

It doesn’t stop it from being waterfall if you try to do something all up-front. Then make changes along the way. Validating against the original up-front design. Dave Farley. Do you really follow iso26262 processes because of the things you work on could harm or kill someone if they don’t work correctly? I don’t think so. And the projects I worked on close to iso26262. I have never been under that. Dave Thomas is absolutely correct here. Processes are ideas. Not concrete facts. But Dave Farley. You often make mention this works for you. You claim ownership. And I love that. But sometimes you back peddle here.

[@ContinuousDelivery]

Not done lots in automotive industries, most of my regulated experience is from finance and medical, and yes we follow those processes. I am not a process expert, but I have worked and successfully applied CD in many regulated orgs and several regulated industries. Mostly, the problem with regulation is the org's response to it, rather than the regulation itself. I always recommend go back to first principles, read the regs, and see if you can interpret it in a different way, don't assume that your org's approach is going to work for CD. Usually in all but one case this works fine with regulated industries, and the CD flavoured alternative worked MUCH better, even from the regulator's perspective. The one case is for what is termed a 'class 3 medical device' that is a medical device that can kill people if it goes wrong. In some places in the world, they require several months of "independent verification by an external 3rd party" before release into clinical service. So we worked around this constraint, following the rules, but optimising for fast feedback where we could, including all of the things that I describe on this channel, with the exception of frequent release into production - we released frequently somewhere else instead. I am not backpedaling, I have worked on safety critical systems, and they are safer when you work with higher-quality, in the ways that I describe here. I don't agree with Dave T that waterfall is every the better choice for software, for other things, sure, but not for software. Still, it was a nice discussion 😉

[@blackjackjester]

Dave (Farley), I highly suggest a small investment in better sound quality for your office/room/studio. Sound production is very clear on your monologue videos...not so much here. Even so though, great series, cant wait for more!

[@PaulSebastianM]

DaveXDave! 😊

[@brownhorsesoftware3605]

🌟 🤩 ⭐ 🌠 💫 ✡ 🌟 Dave squared equals violent agreement! I had to watch twice so I could look up abbreviations. Excellent and enjoyable. Note from a harpist: the conductor is waiting for the orchestra's attention.

[@michaelrstover]

Regarding the dynamic scope and function decorator stuff: if I write a function that expects to be able to call left, right, up, down on the turtle hander, but it's not an interface passed to the function - it's instead invisibly part of the scope my function expects, haven't we made something a bit difficult to use? Or are we talking about compiler/language changes to verify that any caller of the function must provide the handler or be called by something that guarantees it has provided that handler? Does such a language/compiler exist? Also, why is that better than having all these functions that require the handler having the handler in their list of parameters? Other than the fact that we find long parameter lists un-aesthetic? it seems to me, if our function requires something, then it has to declare it so the compiler can enforce it. Whether it's a parameter or an annotation or some other syntactic add on, who cares?

[@Martinit0]

Well I think the logger problem is a more real-world example than the turtle. The common alternatives are either a global logger object/function or passing a reference to the logger to EVERY function that may need it. The third ("handler") alternative seems to have the advantage that you can more precisely limit the scope (compared to global). Though I'm happy to stand corrected, if I misunderstood Dave Thomas.

[@calindinis2050]

Unit tests become more important when it comes to refactoring and code augumentation.

[@petervo224]

16:01 I have to disagree on the advice of "Avoiding Test Nazis". Apparently I am the definition of a Test Nazis, because among all the pet projects I have created, the only one that I can sustain (others have been kept as reference and stale) is the one I keep 2 principles: All Tests pass and Coverage is 100%. Like these are the only 2 principles that you can quickly validate (given that you have the right tools). And when you have these 2, you are free to refactor your codes (cuz your codes will never be perfect and through the change of requirements, you will find it breaks SOLID or other kinds of principles in one way or another). Not only that, you also can change or refactor your tests: Make your test shorter, or easier to read with fluent validation, remove fragile tests, or reorganize them in certain orders, you are free to do it as long as the test set still pass and still cover 100% of codes (of course it does not mean the mutation test is 100%, but to be honest, you never reach 100% mutation coverage). Still, great talk! I love the part there are things you can only learn and cannot be taught.

[@trignals]

What D. Thomas says at 1:17:55 made me think of Reenskaug and Coplien's DCI. Aspirationally, DCI injects behaviour to maximise the ability to reason about code. I haven't tried it so can't comment. D. Farley's association moved in the opposite direction - not injecting essential complexity to improve the ability to reason about code. So I'm really curious if you ever tried DCI D. Farley? If so how was it?

[@BryonLape]

I was in Computer Science program in the late 80's. Had to write Intel Assembly more than 50 lines.

[@jorgec5312]

Hoping you interview Andy Hunt someday too 😊

[@erastvandoren]

01:02:00 I fail to see how this solves the data integrity problem. If two merges intersect, then you have exactly the same problem.

[@ashleydickson62]

Correlation does not equal causation

[@BryonLape]

The Enterprise believes that agile and Scrum are synonymous and have plagued us all with that misguided idea.

[@phoenix-walker]

1:00:00 This take on concurrency and functionally programming and database transactions is inaccurate. You will always have to deal with some level on transactional and concurrency problems, even in FP. User A and User B make a request at the same time to update resource C. Both A and B see resource C at state 1. And make there snapshot of state 2. A functional DB will have to decide which one wins in this case. This FP take makes FP sound like its a silver bullet, and it's clearly not.

[@ChrisAthanas]

50:57 University Computer Science is now considered harmful.

[@DomCim]

I'm not a big fan of the art style of the thumbnails these days. Too much like the uncanny valley.

[@ContinuousDelivery]

Thanks for the feedback, we are experimenting 😉 The idea is that it is supposed to make the thumbnails stand out.

[@roffel06]

It might work a bit better with a more cartoon-ised style? Just a random thought. 😊

[@francis_n]

I think something in the style of the artwork and humour of your t-shirts Dave, I think would stand out well

[@ChrisAthanas]

56:39 yes it’s sad that all this tech was invented in 1968

[@marna_li]

What disappoints the most is that few teans really design their system. Most changes seem to be ad-hoc rather than through through as part of a product. And developers are not really comfortable in making any structural changes even if they need it. You will notice that these are not necessarily the environments with the most agility. They treat changes as a stream of work to be done, rather than something to develop.

[@ContinuousDelivery]

I agree and I think that this is a pretty systemic problem. My observation of most commercial systems that have been around for any amount of time that I see is not so much that they are poorly designed, it is that there is no obvious appearance of design at all. The systemic part of the problem is that, as an industry, we spend lots and lots of time arguing about FP vs OO, Tabs vs Spaces, or React vs Angular and almost NEVER TALK ABOUT DESIGN! This is crazy to me, since the durable skills, the thing that differentiates good programmers from bad, is design - it is really what the job is.

[@marna_li]

@@ContinuousDelivery Yeah. And I enjoy the creative aspect of software development, designing, and learning about the domain and its purpose and importance. Knowing how I can use my technical skills to create value. So to me a use case is important. I have nor met many people in my career who enjoy software development in that way. Few that even have personal projects. I know there are people on the Internet. But most developers that I have met are not “challengers”.

[@cookiejar01]

Of course your project is fine without unit tests in 6 months and coding by yourself. That's like an initial stage for many projects. What about in 6 years on a team of 6 devs that change every 2 years? There's plenty of evidence in industry, it's not like it hasn't been tried and it's just dogmatic. I can tell you because I've seen it many times - it's a total mess! The code can't be refactored, it's not object oriented, nested private methods everywhere, etc. People are afraid to touch any code because any change can break something. Even the smallest change will require extensive E2E testing. It's so bad that it's better to throw away the code and start fresh with unit tests from the start. And if a senior can write decent code without tests a junior in many cases can't. Spaghetti code, not object oriented, etc., etc. I've rarely seen teams of only seniors. Just by making your code testable you fix so many problems.

[@robertkelleher1850]

DT lost me with the state discussion. Sorry but snapshots just kick the can down the road. If you and I have different snapshots that we have changed, we still have to figure out how to merge them back together. Anyone who has ever fried their brain on a git merge conflict knows that pain.

[@banatibor83]

Sounds evil to me to figure out something, write tests for it, and when it is done delete the tests. The next person have to figure it out again, and if he/she is a sensible person, will write tests along the process.

[@trignals]

Thats an interesting point to think about. Imagine instead of deleting the tests you just @Ignore them. Someone who opens up the code can reactivate the tests and whether they work or not the design intent is there. So they behave like comments (allowed to rot silently) until you are actively interested. Reminds me of a comment Elon Musk made about the importance of removing in-process-tests once the (production line) setup is done.

[@biomorphic]

Dave Farley you should really stop promoting TDD and trunk development in your videos. Embracing trens is never a good thing. You have already done enough damage. I have seen so much damage done by the practices you promote, especially TDD and trunk development.

[@llothar68]

Dave Thomas is not a good programmer, don’t think I want take advice from him. Let him publish books and his authors talk

[@cloojure]

Re immutable code: A Maven JAR file is normally immutable, so if you request (Clojure DEPS/CLI style) something like `org.clojure/clojure {:mvn/version "1.11.1"}` you will always get the same library. However, there is alternate syntax `org.clojure/clojure {:git/sha "a8e32aefe55730119040548925cf0eba9ceacb48"}` which pulls source code directly from a Git repo with the specified SHA. In either case, you get immutable code.