Using async generators to stream data in JavaScript

🔗 Playlist for this series
• Iterators in JavaScrip...
🔗Code from the episode
beta.observablehq.com/@mpj/co...
🔗 (Patrons only) Official discussion topic for this episode on the Fun Fun Forum:
www.funfunforum.com/t/async-g...
🔗 Support the show by becoming a Patreon
• Patreon + Fun Fun Foru...
🔗 mpj on Twitter
/ mpjme
🔗 Help translate the show to your language
kzbin.info_cs_p...
We've been looking as async iterators and sync iterators in JavaScript but we've not yet looked at Async Generators, which is honestly what I've been building up to! Async Generators in JavaScript are pretty amazing.

[@tommytomtomtomestini3894]

That "pageIndex = pageIndex +1" line made me vomit a little lol

[@funfunfunction]

I have a bit of a phase where I'm experimenting with avoid the ++ operator. The reason is that I've found that most people don't understand what it actually does. If you ask most programmers what .. let x = 1 let a = x++ let b = ++x ... they'd not be able to answer fast and with confidence. It's not too bad ot use in this case, but I'm less and less concerned about verbosity and more and more concerned about using the simplest tool lately.

[@vidhill]

I heard it was slower also?

[@vidhill]

my info is out of date I'd say, jsperf.com/i-vs-i-i-1

[@mserrano]

Not really bothered by it, but pageIndex += 1 would have been my choice

[@leonsponzoni]

let a = x x++ x++ let b = x "There solved." I like when people separate using from updating.

[@mserrano]

One of my favorite videos so far! This whole iterators and generators series has been extremely useful to me

[@SiriusScaper]

Where I am in the bootcamp I am attending these videos are very valuable to me! Can't wait for higher order async iterables!

[@qwarlockz8017]

Awesome vid! I am going to go back and re watch the ramp up vids for this to get more out of it. Really cool to watch a full process! thanks!

[@DrempDK]

Really hyped about higher order async iterables!

[@uquantum]

Thanks Mattias Petter. I learned a lot. At the end of the journey you take us on, I gazed with amazement at the sleekness of the final async generator code. Like getting to the top of a mountain, completely relaxed while we take in the wonderful view of infinity around us (infinite cat images!)

[@riccardopolacci6501]

Best iterator/generator explanations in the Internet. Thanks!

[@sameedkhan9928]

Good Monday morning! Great video! genuinely enjoying this series and looking forward to higher order async generators. Pretty much opened up my youtube channel to leave this comment.

[@teeodoubled3000]

Great series! Thank you!!

I like to think of the need for iterators as the freedon from the code structure, to peek a "register" whenever I want it: In a loop, your actions are constraint to what you can do between the control line (for/while/do/etc.) and the end of the cycle, while an iterator can be use at any moment in your whole app, and treat each element of the data set in a very free dynamic way.

[@steveneeson5698]

4:13 that sound is killing me :D Thanks, awesome video. What about video around Proxy?

[@ajalbani]

Mpj. I can't thank you enough fir teaching me async generators

[@sebastiendeseille9890]

OMG, it seem so powerful in so few lines. \o/ Yatsa!

[@samirm]

Please do a video on the stream implications you mentioned at the end

[@LarryDiver]

Really cool channel! Thank you, thank you, thank you :)

[@ReviewReactReject]

Hey MPJ great video as always! can you do a vdo on socket io in-depth, please? pls show an example on nodejs! :)

[@jean-baptistedavid6686]

amazing video ! thanks !

[@Zie1u]

What is the editor/plugin that you use for writing the js code and getting a function result next to the line where it's executed?

[@funfunfunction]

It's Quokka: quokka.funfunfunction.com

[@Its.Me007]

Good Monday Morning :)

[@severancestreet]

What's with the `function flickrTagSearch(tag, page)` defined inside the `function flickrTagSearch(tag)`?

[@funfunfunction]

I generally define functions within functions to indicate that they have no business being called from outside the function.

[@filippomenis4438]

Hi Mpj! I work with nodejs, where the concept of streams is already integrated into the language. Since an async generator expresses the concept of streams, does make sense to use it in nodejs since the streams already exists? If yes, when? Congratulations for the videos!

[@amitgujar6231]

even I don't get the exact sense of using a generator. everyone is explaining the hello world program with genrator but nobody showing the exact practical use case.

[@venkateshrajendran3235]

Awesome stuff. Is it possible to use Async Generator with angular data tables. If No, What would be the alternative way to do it.

[@serartmar]

nice videos, thank you) what about continuing "Top 8 developer habits" series?

[@funfunfunction]

Yeah, I probably should. I'm also thinking about starting a series titled "how to constantly start series and never finish them"

[@DucinDev]

hmm, not sure if we can _replace_ rxjs/bacon/etc. with async iterators, becuase Rx streams are push-based and the streams you defined (async iterators) are pull-based. The nature is different. The difference is in who holds the control flow. In pull-based, the owner of the iterator controls how many items can be consumed (how many cat pictures can you see in a period of time in order to get satiscaftion), whereas in push-based the stream is an observable, so flow is inverted (IoC), the platform/lib controls it and invokes the subcriber automatically (the cat pictures change depending on the device and its settings). Both are iterators and both are async, they just differ in who holds the control: you (then it's good that you HAVE it under control), or the platform (IoC, then it's good that you DON'T HAVE to have control) ;) In order to actually _replace_ RxJS with async iterators, you'd have to (1) re-implement all operators (just as rx did), (2) implement observable/observer pattern (just as rx did) so that the stream is lazy as long as nobody subscribed to it. So although you do have the low-level primitives in your hands, you'd have to reimplement the missing higher-level traits.

[@funfunfunction]

Hmm. Maybe. But it might also be like saying that we cannot replace SOAP with REST. Sometimes the constraint and simplicity can be very useful. Rx suffers greatly from it's absolutely absurd surface API area. node.js streams supports both push and pull and are SO hard to grasp. What appeals to me about async generators is that they are very simple atoms.

[@DucinDev]

fully agree with simplicity factor. The best illustration for it, I can think of, are promises - they are very constrained (always eager, only 1 item, non-reusable, access to previous step only, .then which actually _should be_ 3 different methods, etc.) - and that's the key for their ease of use and popularity. However, IoC is less a matter of taste, preferences and just API. It's as if you'd have to check if there is any new post published on a site or checking if an e-commerce product is finally available, instead of subscribing to a newsletter or a product alert. It is _doable_ (to keep on pulling, when the other side should push to you), yet it makes things really cumbersome. IMO it's a big difference - it's not a matter of API, but the mechanics. Anyway, nice recording, as usual :)

[@andreyshedko6155]

Hey MPJ! Currently I'm translating subtitles of this video to Russian and on 0:40 you said "for loop provide iterator"? Am I right? If it's true, that's actually not correct, for loop do not have and cannot have iterator, but its using iterator in order to iterate over collection's items.

[@funfunfunction]

Hmm, no, I don't say that. My wording is a bit dumb though. "We talked about how you can iterate ANYTHING with things such as the for ... of loop because THEY provide an iterator." THEY refers to ANYTHING, not the for...of loop. If I had scripted that sentence I would have written something like... "We talked about how you can iterate over anything that with the for ... of loop as long as that thing provides an iterator"

[@andreyshedko6155]

Thanks, got it.

[@alessiobreviglieri4154]

Video suggestion: Proxy object in JavaScript, Vue 3 implementing his new engine with Proxies and IE slowly diyng will boost the usage of this feature.

[@davidmaceachern7965]

Diggin' the Moderat.

[@Luxcium]

A thing close to infinity 5:55 the JavaScript infinity or the mathematical infinity ?

[@erwinmesi]

MPJ's avatar looks like RDJ XD

[@agorafobicoMusica]

23:10 Maybe you could do yield* pageData

[@100jakkk]

I missed something or you removed your genious intro? :)

[@JoshSandlin]

The `Code from the episode` link 404's :(

[@funfunfunction]

Thank you! Fixed!!

[@nendo502]

just try to add some cache to show the image when fully loaded, the flickering from flickr hurts my eyes XD for await(const url of cats) { const cache = new Image() const img = new Image(150, 150) cache.src = url cache.onload = () => img.src = cache.src yield img }

[@funfunfunction]

We’ll actually be adding exactly this in the next episode, but as a higher-order generator!

[@nendo502]

awesome!!

[@mankindbg]

Mattias you are a fucking genius!

[@thierryhermitte9784]

I'm not really understanding how using iterators differs from using arrays in a stream like you explained, if you created an iterator that is iterating over an array/cache. Thanks, great work btw

[@ratias0]

If the array/list/collection is so big that the RAM available is not enough to handle it, you need an iterator. Why? Well because, after all, it is necessary to load a collection into memory before iterating over it. An iterator is useful if your dataset is 10-20 Gb or even bigger. In fact, with an iterator, you might be free to iterate over an infinite collection.

[@greob]

It looks a lot like Python's iterators / generators (yield) and asyncio (await).

[@DucinDev]

lots of ES6 features were copied (or at least inspired) by python. Another one is destructuring.

[@Dr3amDisturb3r]

I wonder if someone recognized his cat in the video. lol

[@jotaprende]

in what cases in your past, would you liked to use this streams iterators?

[@funfunfunction]

Any time I'd used streams/observables. Node streams, bacon, highland, Rx, knockout

[@jotaprende]

i mean sometime, you wish you had this technology in the past.

[@funfunfunction]

Yeah. I’ve definitely felt that Highland, Rx etc could have benefited from a dedicated syntax and being built into the language. Especially from a debuggability standpoint - stream libraries effectively make you lose the stack trace, so in the same way that es6 promises made promises massively better simply because chrome dev tools added native debugging for them.

[@displayblock6696]

why i got this result? {stat: "fail", code: 100, message: "Invalid API Key (Key not found)"}

[@funfunfunction]

That error message seems rather straightforward, it says exactly what the error is. Did you read the first paragraph that says NOTE in the observable?

[@dctmbu]

can we talk about this wall light? whats going on with Swedens, where did you get this? I want one and become a youtuber :D

[@funfunfunction]

It's a nanoleaf aurora!

[@user-lm7es8mr7s]

By the way, all readable streams in Node.js 10 are async iterators! So we can: for await (let c of fs.createReadStream(filename) { console.log(c) } see: twitter.com/mikeal/status/993925507967205378

[@funfunfunction]

Wow, are they already async iterators? Thats amazing!

[@obedm503]

Now async generators make sense and seem super useful. Before, they where just a new useless feature

[@estephebouvet2147]

To bad you didn't show how to delegate the paging (inner loop) to another async generator with yield* ;-)

[@funfunfunction]

Haha the episode is already 30 minutes long, and you want me to cram more in there?

[@Dalendrion]

On the next episode of FFF: mapping all integer numbers to arrays of URLs with higher order iterables. :P

[@estephebouvet2147]

Knowledge is a never ending quest

[@DucinDev]

I went down that path once and ended up implementing two different streaming libraries... The video would be a killer then!

[@lanogoodman1157]

Am I the only one who does not understand how this `photoindex++ / cache[photoindex] / photoindex=0` thing works?

[@awumsuri]

You are a genius ... When I get $$$ I will become a patron. In the meantime please accept my rainbows and unicorns🌈🦄

[@devpatel6918]

am i alone who think that he looks like tony starck?

[@youjean83]

Why explaining simple things so very comlicated and time consuming? explaination in 1 minute: async function* generator(counter: number) { while(counter < 5 ) { yield await (await fetch(`jsonplaceholder.typicode.com/posts/${counter}`)).json() counter++ } } const gen = generator(1) for await (const item of gen) { console.log(item) } keep the examples short, brief and compact. just my opinion..

[@wilraul9695]

you really miss the point here, he is explaining the concept in a broader way anyways you can limit yourself knowing only how it works.