Streams - FunFunFunction #13

[@okaybenji]

i need at least 3x more FunFunFunction in my life! i look forward to every monday morning now

[@funfunfunction]

+Benji Kay hooray! Thanks :)

[@mishelashala]

+Benji Kay i need at least 3x more FunFunFunction in my life! i look forward to every monday morning now x2

[@5imian]

+Benji Kay This was a really good one. Streams rock.

[@alfaxen1]

Nicely explained. LOVE these videos! For anyone else reading this comment just a tip. If you chain methods like in the video ( at 9:10 , the split, map, map, filter ... ), you should use the .reduce() method, and do all the things you want to do to a signle line in there. The reason is that with reduce, you traverse the object (array / stream/etc. ) just once, where with chaining you do it n times. Its just a performance tip, if you have a small object, the differences are negligible.

[@שמואלדישראלי]

i am a web developer for almost two years now .and obviously i did a lot of studying online a lone and it is hard because of that i am starting a js bootcamp in a few weeks to learn advanced java script development and i must tell you that your videos are excellent, and are a lot beater then some coding books & video lessons & courses i paid for not to mention all the crapy free ones !so great job my friend and as a sign of appreciation & gratitude i will defiantly recommend your channel to all of my friends and collages

[@robbiemilejczak9864]

You seriously rock!! I'm in the processing of learning React and got directed to your videos. I know you're not React specific and React isn't technically functional but there is a lot of overlap and you explain these concepts so damn well and make it so entertaining! Thanks MPJ

[@PeterOeC]

Congratulations! You just earned a new subscriber! (me). :D I ran into RX and streams in form of "Reactive LINQ" (C#) in an Open Source project I am working on at my student job. This is one of the best introductions to streams and RX I've seen :) Now I finally understand the "Everything is a stream" sentence. Thank you for making awesome videos, and keep making them ;) They're fun and very informational! :-) See you in the next video!

[@DanielFlint]

It might also be useful to demonstrate how the transformation "pipeline" is different between streams and arrays. To the uninitiated, at 8:50 the code doesn't look any different to transforming and filtering a plain JavaScript array, so could cause confusion as to why this is better for processing large amounts of data. I think this simple gist shows that difference: gist.github.com/nauzilus/d2e3862162c756ed06b4, and might also help visualise how the data flows through.

[@jaqsudafrican0]

Thanks for that gist

[@felipearosemena8971]

Awesome video! I've been digging into reactive programming and find the observable type to be a leap forward in the land of FE development. I found really useful the way you explained streams and why they are a good thing! Keep up the good work!

[@christopherstephens7278]

I love how you wrote the code and did not edit out the mistakes. Getting to see you reason through and analyze the code was the most valuable information in the video. Would be fun to see you go through some complex code.

[@funfunfunction]

I think so too. The tricky question is - what code could be the target? I can't exactly write an app just for an episode, that would take ages.

[@TheNumberEight]

Thanks! Having a okay understanding of Promises, the Array+Promise analogy at the beginning gave me a framework to think about streams that I didn't have. Like for that alone. Great content mate!

[@funfunfunction]

+TheNumberEight glad the analogy hit home!

[@iAmTheSquidThing]

This is a really interesting programming style that I've never seen before. It's like functional programming in JavaScript. And the indented dot syntax makes it very easy to comprehend.

[@AlliandKyle]

I'm so happy you exist. You've made my life so much easier! You're excellent at explaining this stuff!

[@veeracs]

Amazing videos! Most awesome way to explain the concepts of programming! I wish I had watched these videos as a kid :)

[@cschmitz]

I've been using streams in bash for a long time without really realizing it. I even remember reading stuff about awk and sed (stream editor, duh!) that talked about streams but it never really clicked with me. This video made it click. I write all of this because just now I wanted to figure out if a particular directory path was in my path environment variable so I typed: $ echo $PATH | tr ":" " " | grep composer and realized "OH MY GOD I'VE BEEN USING STREAMS ALL ALONG!!!" Anyway, thanks for this eye opener of a video :)

[@jonacwejman8643]

Hi Mattias! It makes me happy seeing your channel grow! Every time it just gets better and better. I have two questions though. In the first example did intentionally leave spaces after the opening parenthesis on line 10 and 11? And what cozy Atom theme is that? I wish you the best down KZbin lane, och ses i Göteborg!

[@funfunfunction]

+Jona Cwejman Yeah, I like spaces. From a typography standpoint it makes things a bit easier to read. That said, I think that code style is heavily overrated as a means of making code understandable and if you have rules in your linter for spacing, chances are high that you are wasting you very limited time time on things that have very little impact on your quality.

[@lautarowoites6102]

Nice Video. It remembered me about Observables and its operators defined in reactive extensions.

[@kodermine]

Amazing and Explanative Videos! Thank you for making them :)

[@d.sandman]

Awesome show! Sidenote: there is a handy shortcut for `clear` ctrl-L :)

[@CarloRizzante]

:-o That's a little nifty trick, thanks!

[@vitormalencar]

I really like the way the Mattias transforms these " complex " subjects in " fun " subjects

[@renaudtertrais]

The chain of multiple .map() aren't too expensive ? Imagine we had a lot of elements and a lot of transformations, would it not be more preferable to have only one big .map() ? Even if I agree it is clearly more readable and understandable. Excellent video as always !

[@questreal5812]

It won't be too expensive -> they all would be executed one after the other, so the end complexity would still be O(n)

[@polam9799]

oh man, I just started watching these vids after I saw your sabbatical announcement, I bet I can watch them all before you come back :D

[@BadPotat0]

If you got some time, I'll suggest to talk about using a "Message Queue" as well, since they works well with Stream, WebSocket, Web-Workers/threads/async-calls/promises, etc. So you can avoid to "dead-lock" a resource and use it efficiently.

[@MrCiDevant]

Thanks. Great as always. Theres a little error in the first code example :) stupidNumberStream vs. createStupidNumberStream (or was that a test, haha)

[@AnitaSV]

7:55 parseInt(parts[1]) perhaps. Else customers with 13 purchases will be considered less than 2

[@bj97301]

I love these shows. Keep up the good work.

[@daviddesmarais-michaud4345]

This may be a little too simple a question, but why did the split() pipe, split it into 3 parts using the for where to split? In my mind a split with no arguments just puts the value into an array of size 1

[@eugenea8264]

oh, no... no buffering for half string? to glue it to another package into a chunk and then parse full chunk to see EOL symbols? We can end up having 2 strings instead of one in this case... are you sure?

[@evolopterus]

I find streams hard to read/understand visually. Because a stream can go several levels deep, in a short amount of space. Also, it seems difficult to debug. If you get an error in the output, how do you trace back the layers to see where that error has originated? I assume all this is due to inexperience, and it gets much much better with time?

[@funfunfunction]

+Evan Noway (amivag) I'm not sure what you mean in the first part, I think streams are pretty easy to visualizes as they are just a series of magic pipes. But you are right that they are pretty tricky to debug. When I do it I tend to attach "spy" pipes between them to see what values flow through where, but in the end, I think that streaming tooling still has ways to go before it becomes as nice as it could be in theory.

[@simarv0]

+mpjme For debugging rxJava in android, there is a tool called frodo that prints all internal values of the pipe. I don't know about javascript. fernandocejas.com/2015/11/05/debugging-rxjava-on-android/

[@ndstart]

it'll be great if you make a series about coding a project from scratch! and we will all contribute ( code etc ... ) that way i think it'll be so much helpful to all of us right ?

[@EmaSuriano]

Wow dude, you really left your room hahaha! I have such a great time watching your videos, thank u so much for the quality of theirs. Greetings from Argentina :D

[@algorithmengineer6241]

The first example with 2 maps, a filter, and an each looks like an efficiency nightmare! Is javascript/the browser intelligent enough to not run the loop 4 times or will this be a O(4n) operation? I could see that causing problems for large datasets.

[@jesselisser]

Concerning the counter example on the Bacon.js page, is any part of the stream content (possible an array of +1 and -1's?) kept in memory at all? Or ar the +1 and -1's just being applied to the previous number and then forgotten? Thanks for making these... I have learned so much from your videos!

[@funfunfunction]

+Jesse Lisser One of the main features with streams is that you (generally) don't keep things in memory more than you need to - the most common use case for streams is to process data that is too reasonably fit in memory. I'm not sure if that is the case in this particular case (I don't remember anything about this video and I don't have time to watch it at the moment) but that is the case in general.

[@simarv0]

You say that the point is to use tiny composable and reusable units. Isn't there a risk to loose explicitness if the e.g. stream transformations are generalized and declared elsewhere?

[@funfunfunction]

+Simon Arvidsson Definitely. It all depends. In many cases, the parts are so small and reusable that they have already been written and are part of functional languages. In that case, they are well documented and well named and is pretty easy for anyone who has encountered them before. But if course, it's a tradeoff, and one should only generalize when absolutely necessary in my opinion (as I talked about in the straight line code over functions video).

[@oitubo1234]

Very good brief about streams! Mister Fun Fun you rock!

[@ande765a]

Looking at libraries embracing streams and observables (like CycleJS and rxjs) how do you combine the components in a more complex application?

[@ahmedam77]

Very helpful video Thanks for your time

[@challengerssb]

I just found this channel about 2 weeks ago. So many videos, so little time!

[@levisaidmyname]

Hey Mattias, I've been following you on Quora for a while and recently started watching your videos. I really enjoy the content you create and I was wondering if you would consider creating a mini video series on how to build your own NodeJs libraries using c++ ?

[@aion2177]

I still do not understand how split() function works on this example. If someone is thanking the time to enlighten me, that will be much appreciated. I will try to explain where I 'm at. First I want to know for sure if this STREAM we have - is a stream of characters - or a stream of buffers?? For now, i assume that the stream is made out of buffers. Chunks of 1024 Kb memory or whatever is in node. I don't know exactly how much, but I know it's a fixed size. We read this customers file. It's a small file, much lower than 1024 Kb, so we have only one buffer. We convert it - using utf8 - when console.log it - we get one big string which for me is obvious. Why? * because is one single buffer - * the each() method is applied only once - * console.log it's also called only once, and it prints out one string ). - this is what happened in the video before Matthias created the split method. ( see time 6.00 ) kzbin.info/www/bejne/i3WVlY2tbs57ecU But calling split() with no parameters, on a string, will just place that string into an array. Nothing rely happens: For the one buffer we have it's like this: var myString = "Matthias .. "; // the hole buffer transformed to a string using utf8. var theResult = myString.split(); So theResult is: [ "Matthias .. " ] // one array - with one element inside. So how come that on the each() method we get 3 different lines? IF is just one element in the returned array (theResult) - we should get the same thing. One single big string which is the buffer - the first 1024 kb of the customers file. Clearly this is not what happens. So what sort of sorcery is going on here:))? Why the split() method changes the result? Or i made the wrong assumption?! - and the stream is made out of characters? But which doesn't make more sense because we should get the same output like the following. console.log("M"); console.log("a"); console.log("t"); console.log("t"); console.log("h"); When watching the video I feel like I understand everything, but the reality is: I still don't get this fs streams !! Crazy right? If you know better, help me understand this. And Mathias, thanks a lot for this fun, and carefully thought out - high quality content :) This is like the Game of Thrones show for programmers :)). Addicting stuff. I can't wait to see the next episode .

[@hishammajeed531]

+mpjme can you please elaborate how exactly streams do the job of actually flowing the data , i mean how can the script run without having all the thing in it's memory

[@Nerdcoresteve1]

Never understood that Bacon example until just now. :-)

[@phillipvance864]

So are Streams are just values that we deal with Asynchronously? Is the difference between a Stream and a value we get Asynchronously that with streams we have an object that we can call methods on directly whereas with an Asynchronous value (such as a file read with Node's readFile) we must pass it a callback?

[@funfunfunction]

+Phillip Vance that’s a bit like asking what the difference is between a human and a carpenter. There are a bajillion ways of getting values asynchronously and stream is one specific way to do it.

[@funfunfunction]

+Fun Fun Function it sounds a bit that you are actually referring to callbacks now that I’m reading your comment again - in that case, the main difference is that streams can be passed around like a promise, and also can yield more than one value. You might want to think of it a little bit like a promised that isn’t limiters to returning just one value.

[@fabricetreve1180]

Hi, extra thing i would like to know, what camera do you use for filming ? i really really like the grain; thank's a lot'

[@EmmanuelRosani]

Thank you for this! I learn a lot :D

[@lordknighton]

Great Vid. I'll be sure to check out the others.

[@ChurchillLee]

hey Mathias! Can you give us one or two examples of a full fledged open source system that uses functional programming concept (preferably in Javascript) ? I'm new to FP and would really want to see the standards that is followed etc in the FP community. Guidelines would be helpful :D

[@funfunfunction]

+Churchill Lee I think Redux is a very good example of how you can structure things, I recommend looking at that. You might also want to look at Elm, which inspired Redux in the first place.

[@ChurchillLee]

thank you so much :D Your videos have been really helpful to me (watched all of them) I hope you continue doing so !

[@koltongagnon2799]

I've been watching your video's for a while now and I can't get over the amazing terminal emoji's you use !! How can someone customize their terminal to be like that?

[@StingSting844]

One suggestion: scan and reduce do the same but scan returns a value at each iteration whereas reduce returns only the final value. So using reduce on a stream will never return a value since it does not have all values at the time of calling and will only return when the stream ends.

[@deeeeeds]

I know it's a year old at this point, but, why did you go back to using `fs = require('fs')` rather than `import fs from 'fs'`? You've even made a note about this in previous videos. I've only come into JS after the big NodeJS boom, so I'm unsure if there are functional differences.

[@funfunfunction]

Because as of writing, node.js support is still bad for the ES6 style imports.

[@jabano3]

Im having troubles understanding the diffrence between using var, const and let, care to elaborate? ps: loving the channel

[@joemethven5925]

Whats the name of the theme you use on your editor? Love the colour scheme!

[@fsacer]

github.com/wesbos/cobalt2-atom

[@danangyudhatama]

why the line 4 works ? i think .split() will return all of the text in one element of array. Should it be .split(" ") ?

[@funfunfunction]

It splits by new line per default. See highlandjs.org/#split

[@danangyudhatama]

thanks a lot for answering my question mpj, although this video was published 2 years ago

[@Overthought7]

Huh? I still have no idea what streams are, or why I'd use them. Your usage example of a large file seems odd to me. Can't you just make a file handle and read one line at a time? And also, isn't the data supposed to leave the stream? Like if things are just entering the stream (like the up/down example), isn't that equivalent to simply appending to an array? Nothing fancy required there.

[@KDOERAK]

great explanation!

[@maximilianlloyd2111]

How do i solve it if i need to hit a JSON api, and then process each of the objects, and this API will be huge, perhaps several GIGs.

[@funfunfunction]

Use www.npmjs.com/package/stream-json

[@MarcelRobitaille]

Wow. Streams are nice. Great video.

[@toefour]

3:52 King Mukla, Hearthstone/Warcraft reference?

[@funfunfunction]

+Chris Stevens Well caught!

[@romebop]

how are you able to run the arrow functions on node without the harmony flag?

[@funfunfunction]

I use the latest node.

[@vyacheslavpalamar1787]

Hey, MPJ! Please, can you give me an advice? I started to learn JavaScript, but in general I've learned programming for the last year. But I don't feel like I made much progress. I still watch tutorials and solve problems at codewars. I don't have any idea about "real life" project. So, can you tell me, how can I improve my skills and reach new heights?

[@sentienceRemains]

How’s it gone?

[@vyacheslavpalamar1787]

@@sentienceRemains wow, thanks for asking, I have been working as a software engineer for almost five years actually! What a journey 😅

[@sentienceRemains]

@@vyacheslavpalamar1787 that’s amazing! Way to stick with it. Here’s to hoping for similar progress!

[@drewdeal1632]

This will help me train folks in CycleJS!

[@indigo0086]

@mpjme what's your setup, computer, terminal, editor, etc?

[@Jliuify]

and how did you get the farting sheep on your terminal CLI?

[@Brunoenribeiro]

Fantastic channel, man :)

[@pedropinto8465]

at 9:19.. in front then programming?

[@danielchavolla1507]

This stuff is so good it feels that i'm supposed to be paying for it.

[@StephanHoyer]

Again, great. Logical follow up would be generators I think.

[@yoelmonzon6992]

How do you set the goat in your terminal ?

[@funfunfunction]

+Yoel Monzon Add this to your .bash_profile: PS1='\W 🐑 💨 '

[@yoelmonzon6992]

Thanks, it's so pretty 🐑

[@christopherbarber7705]

thank you

[@TimothyWhiteheadzm]

If your stream returned a single string which you then used .split() on, then surely from there on it wasn't a stream, and you got none of the benefits of streams. ie you had to load your full 10Gb file before you even called the first .split().

[@coolworx]

+Timothy Whitehead That was a WTF moment for me as well...

[@DarkZeroVenarius]

from what I see .split() is actually a function of highland, when I did a test, using a big file and making console output when mapping or filtering is being done, it was very obvious that the lines basically flow through the structure line by line, I think highland is working some magic here :)

[@rickytruesdale6646]

you are gay

[@autochton]

It's what's called a chunking or buffering function. A stream can come at you in chunks of some usually ill defined size. A network data stream, for example, where you're getting data in from some other machine somewhere out in the -great blue yonder- Internet, you might get it chunked by network packet, which may be too much or too little for you to work with. Let's say you get packets containing 256 bytes, but you have several-kilobyte lines of text. You then want to chunk your stream differently - and what that does is basically take in the stream data as it arrives, and look for the criteria for a chunk border, like a carriage return for line chunking. (Might be any of a million other criteria, too, whatever you need.) Until it hits one of those, it buffers up the stream data, and then when a chunk border arrives, it emits the whole chunk as a unit instead. So instead of a stream with chunks like this: , you get one with chunks like this: Make sense?

[@przemysawlib4309]

.split() will work on singe chunks as they come it return stream where each value is line. That stream is actually subscribing to original stream, and it works on one chunk at a time.

[@1DJLNR]

At 1.43 i almost died of laugher.

[@AndersRapp]

Awesome t-shirt! :-D

[@IsraelBautistaWebDev]

I know almost each month there are new code schools or bootcamps everywhere promising to teach you to be a programmer in "x" amount of time, but have you consider in the future to create an online class business? Since you're already doing videos anyway and you have a natural talent to approach teaching in a non-intimidating way.

[@funfunfunction]

+Israel Bautista Online classes is a possibility, but I kind of lack the attention span to make one, I think. We'll see.

[@LeonardoFrangelli]

Very nice!

[@alexalexson7556]

Your videos are great but I'm still confused what streams are. Thank you for igniting my curiosity.

[@funfunfunction]

you might want to check out my monad talk from nordicjs, that talks a lot about them too

[@adityahalabe5482]

You can think of Stream as baby of Promise and Array ...

[@tactical.precision.disarray]

Nice shirt, man!

[@hamesjetfield0]

05:07 "Data and functions are separate" Tell that to a lisp programmer ;)

[@WambaTube]

That's cool.

[@PhillipSenn]

@1:43 scary

[@NikolaiAleksandrenko]

More please :)

[@Mrgreatestfreakout]

i like your videos lol

[@Denzilb55]

Do you accept bitcoin donations?

[@sentienceRemains]

He should’ve

[@OktayAcikalin]

I just paused the video at 1:10 :)

[@funfunfunction]

gold star!!!

[@silasalfredston5511]

Good good good good good good good good good! Thank you :-)

[@nanthilrodriguez]

GMT Time. That made me lol hard

[@JasonCtutorials]

wow you watch adventure time. Dude!

[@engelshentenawy]

don't be fooled streams are complicated but fun ... sorta

[@Textras]

2019 consider Most.js and Kefir

[@zexsen85]

It sounds lik a currying :)

[@karlpokus]

Vafan?! Jag pausade!!

[@tanmaydas]

Like your videos. Hate the background music. Frankly, background music does not help a learner in any way. In fact it hinders the learning process.

[@gab229]

J

[@venicebeachsurfer]

RXJS

[@funfunfunction]

+Emanon it unfortunately also has the surface API of Sahara. Streams is not a complicated subject, but RxJS makes it unnecessarily hard to deal with just because of its vastness. I hope that we can get a simple observable construct built into JavaScript soon.

[@rajington]

I'm convinced your quest against code verbosity is because you're so bad at typing... :)