Pipeline-oriented programming - Scott Wlaschin - NDC Porto 2023

[@TechTalksWeekly]

Scott's talks are marvelous. Here's a list of all his best talks so far ordered by the number of views: 1. "The Functional Programmer's Toolkit - Scott Wlaschin" - kzbin.info/www/bejne/hKPTkH-QYr16qbs 2. "The lazy programmer's guide to writing thousands of tests - Scott Wlaschin" - kzbin.info/www/bejne/f4rddXl-rZuEhrs 3. "Four Languages from Forty Years Ago - Scott Wlaschin" - kzbin.info/www/bejne/ZpfTdZ93eracpZY 4. "The Power of Composition - Scott Wlaschin" - kzbin.info/www/bejne/rHXIXmelbbqqopo 5. "Pipeline-oriented programming - Scott Wlaschin - NDC Porto 2023" - kzbin.info/www/bejne/n6HGloerf9GqY68 6. "Reinventing the Transaction Script - Scott Wlaschin" - kzbin.info/www/bejne/i4S2nJyaosaGbNk 7. "Thirteen ways of looking at a Turtle - Scott Wlaschin - NDC London 2021" - kzbin.info/www/bejne/nWGZmpqhrcd5bZo 8. "Building confidence in concurrent code with a model checker - Scott Wlaschin - NDC Oslo 2020" - kzbin.info/www/bejne/qqLalK1jjtlsnbM btw. I'm building a newsletter called Tech Talks Weekly (techtalksweekly.substack.com/) where once a week I send out all the recently uploaded tech conference talks across +100 engineering conferences including NDC, GOTO, Devoxx and more. Consider subscribing if this sounds useful.

[@TechTalksWeekly]

btw. I highly recommend his book called Domain Modeling Made Functional where he presents Functional Programming as a perfect paradigm for doing DDD.

[@joaomonteiro6245]

I subscribed. Also added all of those in a playlist kzbin.info/aero/PLdbbFSJJ_c_RC2c5_te9aKtS7B3BdQD0J&si=AdqVlojwkehvs-Be

[@TechTalksWeekly]

@@joaomonteiro6245 Great idea with the playlist, thanks for putting this together! And also thanks for subscribing :)

[@joaomonteiro6245]

@@TechTalksWeekly no, thank You for putting the links together and the Newsletter!

[@kberestov]

Any Scott Wlaschin's speech is a quality mark. Thanks!

[@gdargdar91]

A presentation by Scott Wlaschin - instant click!

[@themcchuck8400]

"Pipeline-oriented programming" is more properly termed "concatenative programming". There is an entire family of languages devoted to it, namely Forth, Joy, Factor, and Listack. There is even a concatenative hack of Clojure called Gershwin. Manfred von Thun did a lot of the theoretical work on this programming style 20 years ago. It's really the ultimate extension of function composition. The basic principle is simple - you send data through functions in the order in which they act upon the data. (As opposed to the usual inside-out or backwards function call style.)

[@Verber0n]

Or dataflow programming.

[@maxymajzr]

I'd like to stick to pipeline as terminology. Thanks to both of commentators for invaluable input as to what "proper" might be, but pipeline works just great so if you don't mind - I'll use that one. Cheers.

[@chudchadanstud]

Also Functional Programming

[@artxiom]

Or functional reactive programming. There are so many names for it. Even monads do that, they are "pipeline-oriented", and they solve, what a coincidence, the issue of IO in pure functional languages such as Haskell.

[@VincentDBlair]

Per usual, thanks Scott.

[@nexovec]

To me, in the narrow realm of software engineering, systems and software design are the two of the most sexiest areas ever. The best part is it will never get old or out of fashion.

[@lfarrocodev]

I've been using this pattern for some years. Data train goes from station from station, no side effects. Just accumulating operations as data (makes testing very easy is well). On some stations, some choo-choo is performed (fire the side-effects) and then the train continues its journey...

[@satvistayou]

Choice / Choose - the input can only be one of the possible options. We compose all options and a single choice is made - Sum types / Sealed classes Seq / Ordered conditions - we compose an ordered set of options and process input in order and choose one that matches - Routing And All - we break down complex conditions into set of simple ones. The input has to satisfy all of them Sequential transformations- we process the input via a series of transformations. Each one can be conditional and successive ones can do retransformation. Parallel transformations- do the transformations in parallel. Might require a combine or reduce operation at the end

[@artxiom]

I've been programming like this for a few years now and it has been a game changer. Being able to reason about pure logic and complicated runtime state changes separately just scales so much better, is easier to debug, maintain, understand, etc. This is btw. implemented in frameworks such as Redux, ELM (view-update-model), FRP and similar. I would argue that even React works like that: you have pure, declarative components, event handlers and state management. While they can be kind of put together they are conceptually separated. The idea is always the same: you have a functional/pure part separated from the IO part. They all only disagree on, or have a different vision of, how to do this exactly. Actually, this pattern is very widespread when I think about it - seems the entire industry silently embraced it without many people even noticing. My theory is anyway that sooner or later FP will dominate development. There is hardly any mainstream toolkit, language or framework that doesn't include at least a few FP-concepts - and it's getting more.

Amazing presentation.

[@manualautomaton]

I'm a fan of fluent interfaces as well as functional programming styles, but when it comes to pipelining, I'd love to see an example of connecting functions with more complex signatures. that can get tricky..

[@codeman99-dev]

What exactly can get tricky? "More complex signatures"? What does that mean? F# arguments are always curried. A let binding can hold any data or partially applied function.

[@kbaafi]

This is an OO answer, but you can use the Command pattern to encapsulate all the details of an operation, thus making it possible to build a chain of commands. The output of a command could be used as input to the next command and so on

[@Uerdue]

This is actually quite a common architecture when building compilers.

[@SaudBako]

This is just layered architecture, which can be done with OOP or bricks, or even stones. If one tends to spaghettify their objects, they'll definitely do the same for functions.

[@7th_CAV_Trooper]

Hmm. This is what I've been doing for years, but I've been calling it OOP. I probably started building like this around the time I discovered TDD or Clean. What a great talk.

[@magne6049]

52:15 It’d be nice if it was possible to use React in this way. React being the pure code parts (explicitly calling React.render) and I/O being user input or network requests. Either you could have all fetches in the top level, respecting the purity of this Onion architecture (aka. ‘functional core, imperative shell’). Alternatively, the React.render call could collect all side-effects (like fetches) and then you could batch them and execute them in the I/O step. Like a structured concurrency approach with EffectJS. It would solve the fetch waterfall problem.

[@magne6049]

maybe you could co-locate fetches in XState machines next to the components, but pass that machine down from the top of the page, so it is always called (with all fetches for each component on the page) on page load?

[@16M-w4y]

A nice approach to software development and coding 👨‍💻😎, but I find it somewhat similar to the middleware and loop pipe approach that was introduced in .NET Core 3.2, and also bears resemblance to the Aspect-Oriented Programming approach that focuses on the business management layer, so I see it as taking an advantage from AOP. And an advantage of middleware, and the result was this beautiful approach. I do not deny that it contains its own features, but the basic things that represent the essence of this approach are what I mentioned previously. Thank you, teacher, for this nice lecture.

[@tilentratnjek8743]

How is this diffrent form the fluent interface design pattern?

[@georgepagotelis]

It’s the same but the pipe generic is interesting addition.

[@ks1970in]

can we overload operator | in C#

[@coder_one]

Didn't Scott originally use the name "Railway-oriented-programming" instead of "Pipeline-oriented-programming"?

[@mitzrael2k6]

Railway is for error handling.

[@WorstDeveloper]

Interesting, but how do you break a pipeline if the remaining functions should not be executed? With a for loop you can break or return early.

[@dmsoares]

I suspect you'll get the answer on the other talks. There were several examples of this in the simple web server: GET fails if the request is not valid and the following functions (route and setStatusCode) are not executed.

[@kostasgkoutis8534]

Mix it with the Result pattern

[@martinfreund3007]

basically you prepare each function to short-cut. See more on this in railway-oriented programming: kzbin.info/www/bejne/nIrSZH-EbruZla8

[@IsmeGenius]

He showed it in the webapp example, it uses maybe monad without saying a word monad and making railway analogy instead.

[@Samedhi1]

Is this any different than Clojure's thread[first|last|as] macro with a sequence?

[@aguilaaudax1362]

actually functional programming

[@halfsourlizard9319]

OOP is so weird: Why would you imprison your functions in 'objects' (which are just weird self-referential mutable records)?

[@thirdvect0r]

No one's talking about the fact that other the course of the talk he managed to sing the lyrics to "Never gonna give you up" by Rock Astley? We got Rick-rolled in an NDC talk :D

[@arieheinrich3457]

fantastic talk !

[@chicoern]

PowerShell!

[@vram288]

at 32 good

[@hhhhhhnnnnnnnnnnnnnn]

amazing its the same talk he has been giving the last 5 years.. NOTHING NEW!!! Read his book (years old by now).. same content..

[@magne6049]

education never stops, there are always more people and new generations.

[@artxiom]

Yeah, because these are timeless concepts.

[@chudchadanstud]

Aren't these just called streams?

[@iwhitt9484]

Great talk but Windows let him down, switch to Linux and say bye to your blue screens

[@halfsourlizard9319]

GNU/Linux is vastly more stable and doesn't require random reboots. Skill issue if that's not what you're seeing.

[@raptoress6131]

Just when I thought that I need to learn more about pipelines 😊

[@MCLastUsername]

This appears to be nothing more than pure functions plus method chaining, which is indeed a useful idiom, but it's hardly a "paradigm". I don't see how it's more composable or testable than pure functions in general, nor do I see how it is an alternative to OOP. Actually, in most languages, this is only possible thanks to OO method call syntax.

[@simonfarre4907]

I think you are wrong. He just misnames it. What he is talking about seems like DDD, but he for some reason calls it pipeline instead.

[@lorenkuhn3806]

Was thinking the same. We do this all the time in pure functional code. there is literally no other way. He most likely just tries to find a less intimidating name for functional programming. For whatever his reason is...

[@chewbaccarampage]

I think this talk is really great for people who are unfamiliar with FP. It highlights the benefits of composition without putting FP in the forefront. I still work with "senior" developers that refuse to learn or use LINQ based solely on opinion and dogmatic principles. Giving it a new name might make those people tune in.

[@dmsoares]

Yeah, I actually give Scott a lot of merit for simplifying FP concepts and design for all the tech leads and CTOs out there that wouldn't otherwise sit for even 5mins to listen to an FP talk, unfortunately for them and for all the developers actually interested in finding better ways to write composable code

[@artxiom]

Yeah, it's just FP explained to non-FP programmers. But then also why do we even need an OOP-alternative? To this day no one could explain to me what kind of problem it really solves. > Actually, in most languages, this is only possible thanks to OO method call syntax. No. How so? In most languages the OO method call syntax is just syntatic sugar for function(self/this, ....). Even virtual functions are nothing more than fancier function pointers. In C++ for example a class is pretty much equivalent to a struct (the only difference is that class members in a struct are public by default and private in a class). Inheritance can be easily replaced by composition (which is superior anyway). There is absolutely nothing(!) that can be solved easier with OOP than without. It's just a non-sensical abstraction put on top. Not only did it's inventor (Alan Kay) had only a vague idea what it should actually be, no he also admitted that he didn't understand FP and said that OO was wrongly implemented in pretty much every modern language (at least not according to his idea). It was a huge mistake, but companies invested billions in it (Java *cough* *cough*) so many don't want to admit it.

[@ivankocienski1]

"Grokking Simplicity" (2021, Eric Normand) is a more thorough introduction to Functional Programming (also the book crashes a lot less than this guys laptop, yeesh windows)

[@artxiom]

Hey, it's not his fault that Windows isn't FP and pipeline-oriented ;)

[@dlabor1965]

Guru Meditation...

[@atari1040]

Wow - just like core idea in UNIX in 1975.... LOL! When world is going to completely reject this madness with OOP?

[@raptoress6131]

When Java dies... 😅

[@МаксимГорюнов-м7и]

​@@raptoress6131 which is not happening

[@SteveSuehs-c5b]

That process has started. It won't be a complete rejection, but with 'Out of the Tar Pit', OO-everwhere began its descent.

[@chudchadanstud]

OOP is good

[@Alexander.Glazkov]

Компьютер перезагружался потому, что народ охренивал от того, что он снова в очередной раз рассказывает одно и тоже! Сколько можно уже одно и тоже рассказывать-то? Может уже пора на пенсию?!

[@georgepagotelis]

😂

[@chudchadanstud]

This is just OOP. The arrows are the class data the boxes are methods. The good thing about OOP is scope. The data in the arrow is private