Function Iterators might just change the way we write loops in Go

[@johnmarianhoffman]

I think a lot of the potential harm that could be caused by this feature would be almost entirely mitigated with a strong standard library implementation of many/most of the most common use cases. I think this approach is fine, and generally needed when writing a lot of iterator-heavy code. This seems fairly reasonable. What I get scared of is every sophomore programmer thinking they need their own implementation of exotic iterators, writing their own package, and it just creating a bunch more headache that would've been prevented with a simpler or more explicit approach. Not everyone should get equal access to parallel iterators! That comes with some very real risks!😆

[@sevenmore3355]

This feature sounds more and more like templates in c++

[@kenneth_romero]

also couldn't he just put the code in the for loop in its own function to make the logic more readable? seems like so much work to create all that code to have an iteratable function

[@kevinb1594]

The waitgroup verison of the example is MUCH easier to reason about. I would hate to have to debug that function.

[@IamI16]

me too

[@pmartin]

If the function is implemented by a random CoWorker, with a silly name, obscur pattern etc. Yeah obviously, I prefer the WaitGroup version. But, If Parallel becomes an actual thing in std, I'll be happy to use it. It's like Generic, random package using Generic for nothing or container that I never needed, is a redflag for me. But, Generic helpers like `cmp.Or` or the `slices` package is actually a great thing which remove a lot of boilerplate.

[@andrewtran9870]

​@pmartin I agree with you on this, seeing stuff like this in random codebases are a nightmare to debug. Placing it in std, devs can get accustomed with the behaviour. However, for a language that strives for simplicity, adding features like this can lead to bloat and complicate learning the language. Personally, I don't think that's too much of a problem here. Additionally, because std library lacks things that reduce boilerplate (like this), devs are more inclined to add this type of code into the codebase themselves. Essentially creating their own utils that vary slightly between codebases and can be annoying

[@DerTim]

Nice feature to extend loops, but such an implementation feels difficult to handle and violates the value of simplification. So, whatever the Go Community prefers :)

[@TheQxY]

The syntax is not the most readable, but overall, I'm in favour of the proposal. It will be a very useful tool for the Go devs and for library devs like myself. Endusers don't have to deal with the inner complexity as much and can have a more homegeneous experience dealing with ranges. It's up to the library devs to write self-documenting and readable iterators.

[@lulzmachineify]

Everyone should be able to do be "library devs" and not just "end users". The beauty of go is that things are readable, maintainable and easy to just jump in and build features in. Creating this kind of separation between "code that everyone can work with" and "vodoo magic, here be dragons"-code will hurt everyone

[@davidgillies620]

Python and C++ (modern C++, anyway) make fairly heavy use of generators (in C++ up to 20 they were implemented with input iterators; C++23 has an explicit std::generator class). It's a very powerful pattern for parsing data streams, for example. Node has had something similar since basically forever.

[@coffeeintocode]

This is really powerful. But it feels very un-Go-y……eg it’s too much magic. Obfuscation isn’t good for Go, I know verbosity gives it a bad repo, but I also think this forces better code

[@l_unchtime]

That's why it's experimental, and will improve over time :)

[@jordanwright3052]

​@@l_unchtime 😂🤣😂🤣, u sure about that?

[@JonasWeckschmied]

I'm genuinely curious why you consider this "magic"? The only thing that changed is that range can take functions with a specific signature. No new keywords that complicate the language, just something you can do if you really need the power of iterators.

[@MarisaClardy]

Nice to see Go finally getting generators. I use this pattern a lot when working in PHP, so good to see it's going to be possible to do it in Go.

[@daviddesmarais-michaud4345]

This video is great. The explanation and animation is great. One nit, the Parallel implementation is wrong. It works because it is unlikely that that race condition will be met once you've canceled the context, but another goroutine could be yielding in as you cancel. This is a classic case where you need a mutex. A potential solution could look like this: func Parallel[T any](list []T) func(func(int, T) bool) { return func(yield func(int, T) bool) { var ( wg sync.WaitGroup m sync.Mutex cancel = make(chan struct{}) isCanceled bool ) wg.Add(len(list)) for i, e := range list { go func() { defer wg.Done() select { case

[@what1heh3ck]

but it will no longer run in parallel right?

[@daviddesmarais-michaud4345]

@@what1heh3ck well this example is trivial. Technically you would want to be doing doing some work to generate the items you yield. In this case we are doing no work, and since we need to synchronize at the moment of yielding as per the spec or risk a panic, then in this case there is no advantage to using concurrency and only serious drawbacks. But as an example of how concurrency could be built into function iterators it’s great! TLDR: we don’t want concurrent yielding, but concurrent generation/work.

[@TheArtikae]

I’m not super familiar with Go, but isn’t your code holding the lock while it calls yield? Yield being the function that runs the body of the for loop, right? The entire point of this wrapper is to run the body of the for loop multiple times in parallel. Your function doesn’t do that, as far as I can tell. All your version seems to do is run a single-threaded for loop split across N threads. I’m also not clear on what race condition you’re trying to avoid. The entire purpose of this wrapper is to call yield from multiple threads simultaneously, so that can’t be it. I suppose there is the case where one thread calls cancel while another thread is in the middle of calling yield. However, that’s an unavoidable consequence of the task at hand. If you run a for loop in parallel, you don’t get any guarantees about iteration order. Break could never work like it normally does.

[@daviddesmarais-michaud4345]

@@TheArtikae you are correct. However the code shown in the video is not safe. The context does not definitely block two goroutines from yield racing whilst one returns false. I am not sure this parrallel can be safe in Go using function iterators. Mine basically just made it serial

[@TheArtikae]

@@daviddesmarais-michaud4345 Oh dear, I assumed that yield was safe to call from multiple threads, seeing as that was the entire point of the Parallel wrapper shown in the video. I was wrong. You're absolutely right about the issue, and the data race. The wrapper could never possibly be correct with function iterators.

[@yakomisar]

Elliott, pls, make a step by step guide on how to use pprof professionally.

[@flannn6]

THIS! cpu and memory profiling.

[@wodxgod]

yesss

[@plaintext7288]

This will be very nice to work with when using remote data sources - just treating them as arrays, imstead of something like 'for val, err := source.Next(); err == nil { dostuff(val) }

[@niksaysit]

The syntax of "functions returning functions accepting functions that you can pull values from" is too complicated and magic-y. It seems like they're trying to stretch the extra-simplistic syntax of Go1, when it might be better to learn from its shortcomings to come up with Go2 But that's my humble opinion

[@ForeverZer0]

They try a little *too* hard to never add any new keywords or core types to the language, and always reuse what already exists, in this case "for..range". Sometimes it would be better to just expand the language a bit for the sake of clarity. Reading the developer discussions on various language proposals is a frustrating experience: practically everything is rejected if would require anything new whatsoever to the syntax of current lexicon/vocabulary of the language. I appreciate that they err on the side of simplicity, it is one of Go's greatest strengths, but then there is examples such as this, where it actually makes the language more complicated by being so strict.

[@salamander1782]

I highly prefer the boiler plate "heavy" way of doing this. No reason to abstract away easy patterns everyone knows

[@Amejonah]

Every time I'm surprised by the fact that go hasn't all the things I take for granted now in Rust and Kotlin. I need to make my own stuff too much, it feels like left-pad all over again.

[@IgoR.R.]

This experimental feature challenges boilerplate oriented programming. Jokes aside. It's more convenient for sure. In my opinion it's bad for go's simplicity.

[@ForeverZer0]

yeah, it does feel like a little out-of-place for go. This would perhaps be a good candidate for a dedicated keyword (i.e. "iter" instead of reusing "range") to make it abundantly clear what is going on, and/or some refinement on the syntax, which feels a bit clunky. While i understand it is not a requirement to write it as such, realistically the syntax encourages that this is most often going to be written as layers of anonymous nested closures, which feels "off" as a core language feature.

[@IgoR.R.]

​@@ForeverZer0 Dedicated keyword could be too confusing. It's feels weird that range for an array or a slice yields index and value, but just value for functions. Take python, for example, it has "for x in iterable". It doesn't matter if iterable is a generator or an array. But definitely less confusing because of consistency.

[@wonderful-p7z]

it seems rust is becoming easier and clear than go. The only good thing i like about go is compile time.

[@omgnifty5957]

As someone who stills prefers rust... go has way, way, way easier concurrency. Doing async stuff in rust is a huge headache if you go beyond trivial examples.

[@funky_hedgehog]

@@omgnifty5957 In Go you will get panic in production. In Rust compilation error.

[@TheArtikae]

This is just try_for_each tho, isn’t it? What’s the big deal?

[@sdramare864]

@@omgnifty5957why can't you use tokio select and mpsc channels to achieve the same concurrency approach?

[@Redyf]

Go is such a cool language, I just started learning it so this was very helpful. Thanks Dreams!

[@markhaus]

So long as the standard library gets a good set of iterators covering most use cases this could be quite good without introducing too much magic you need to lookup to the language. My favorite thing about go is it’s dead simple and you can at a glance see exactly what the code is doing.

[@zarbis]

5:40 that moment when I'm finally stupid enough for Go.

[@bijayaprasadkuikel5162]

Please create a premium course on Go. I am all ready to subscribe to it. 🎉🎉🎉🎉🎉You are probably the best teacher who teaches golang perfectly.

[@PawelKraszewski]

So it took them just 15 years to discover iterator infrastructure... Now we wait for local-scope level for defer (that is calling on end of the current scope, not the whole function), mutual module inclusion, configurable ciphersuites in TLS1.3, modern error handling and a few others.

[@dreamsofcode]

Another 15 years for null safety!

[@tomas120]

@@dreamsofcode is null safety really needed? I mean, c++ doesn't have it and also you have to deal with null pointers. I think Go was designed with that kept in mind

[@cylian8422]

The thing I'm waiting for the most is sane enums. Just do what Rust is doing in this context

[@PawelKraszewski]

@@dreamsofcode Null safety? I'm merely asking for const ptrs... func (hs *const HugeStruct) CantDamageHS() {}

[@PawelKraszewski]

@@tomas120 C++ has references that guarantee null-safety (if not abusing some UBs), if you fear pointers.

[@ev3rybodygets177]

yaaa.... it think go should bench this one. it just seems like the plot has been lost in this feature. i know go wants to be this gc lang that has first class support for concurrency to make up for that but it also promises to stay simple like c. It feels like too big of a sacrifice of the former in favor for the latter....

[@pdougall1]

Maybe it's the specific example, but this implementation made the code far more complicated and less obvious than it was when using the async and go func directly. It can be bad when abused.

[@BosonCollider]

I mean, the video specifically intended to show a footgun example. There's other examples that swing the other way like iterating over the results of postgres cursor without leaking it. This makes it possible to write code that will not leak resources needed by an iterator (unless you use iter.pull)

[@Fudmottin]

Parallel processing can definitely get interesting when hidden somewhere down in code you don't know about there is a shared resource. Something like a memory allocator perhaps.

[@guitaripod]

As it stands, I'm not sure I like it. Go's power comes from the verbosity, and obfuscating that behind a convenience interface seems more like other languages for little benefit

[@dreamsofcode]

Yeah agreed. I'm in two minds about it. I think for the standard library to tidy up the many iterators patterns we currently have is a good thing, but for concurrency it might hide too much.

[@AmirHosseinHonardust]

Have you ever used them? There is not much hidden. All the logic behind these functions are obvious.

[@ivyZorz]

So it seems like function integrators are, generally, a good feature. The concern seems to come if an iterator implements concurrency because the concurrent logic is obfuscated behind the function. I’d love to see some high level iterators implemented by the std lib but default (or only) be synchronous. Maybe there could be a pattern to “enable” or “opt in” to concurrency on the new iterators. That way it would be obvious that they’re concurrent without having to read how exactly that iterator you’re using works

[@qlcorp]

This is an affrontment to my sensibilities.

[@flannn6]

very interesting! great quality content as always. thanks

[@Antonio-yy2ec]

Just what I needed! TY TY TY

[@sdramare864]

Finally Go has gotten a feature that existed in other languages like Python or C# 20 years ago.

[@DM78]

One of the ways I've been identifying goroutine usages obfuscated by a function call is to require a context passed in as the first parameter.

[@nubunto]

Yes, this video made me appreciate Rust even more. Thanks!

[@MrlegendOr]

Rust is trash

[@nubunto]

L

[@heldim92]

I know nothing about neither Rust nor Go, but this code seemed complex to me, maybe because I have zero contact with the language. My question is, when a person gets to the point where they need to use such functionalities as shown in the video, is Rust a more practical language when it comes to achieving the same results?

[@nubunto]

@@heldim92 I'm not saying it's _practical_, I believe that would just be untrue (depending on your definition of "practical"). But what I came to enjoy about Rust is that code is explicit: you know at compile time that it is race condition free (at least in your own usage, i.e. multithreaded code) , mutation-free, or explicitly signaled as a mutation. Is it more practical? Most of the time, no. But it's usually correct. And iterators are already baked into the language and a joy to use

[@simonkalu]

@@heldim92 For me Rust is more cleaner, logical and readable than Go

[@Dissolve123]

what do you use to animate the code in your videos?

[@nikmy_]

1. Goroutine G starts iteration, fall into default section in select and runs yield function 2. Goroutine H starts iteration and fall into default section in select 3. Goroutine G receives false from yield function and cancels context 4. Goroutine H call yield function and must panic, because it is yield after break Parallel does not work :( To fix that, we need blocking yield implementation, and await on yield channel in select

[@random-staff]

Was: simple & explicit, will: tricky & implicit

[@_hatred]

if go manages to polish typing system or move to traits (which is unlikely), it'll be hands down the best language

[@antonbozhinov]

So basically to summarise…. Its hard to read 3-4 lines of extra code as you miss the intended logic. Therefore we introduce another layer of abstraction with indirection and unintuitive usage… but hey… we can read 3 lines less now… 😂

[@recarsion]

I think this could be really powerful but it's kind of complicated, I would use it sparingly and wouldn't introduce custom iterators to a project if I didn't have to. Old-school for loops and indexing can be cumbersome at times but even a monkey understands what it's doing.

[@idearat0]

Love the feature overall. Perhaps one option for the async question would be to prefix all functions that involve async with Async, similar to the convention of 'Must' for functions that panic?

[@0xngmi]

the less boilerplate the better IMO. makes it easier to understand and to learn

[@matiasbpg]

It would help if static analysis tools warned about concurrent iterator functions. I see no harm with sync ones

[@dreamsofcode]

They've actually added a fix in for 1.23! Now it throws a panic when it detects concurrency behavior.

[@matiasbpg]

@@dreamsofcode Nice! But then there is no async seq? Will it have another signature?

[@dbalencar]

How to configure vs code so it doesn't show syntax "error" when writing iterators for rangefuncs? For running I know it's just GOEXPERIMENT=rangefunc .

[@dbalencar]

I'm getting: cannot range over backwards(nums) (value of type func(func(i int, x int) bool)) compiler (InvalidRangeExpr)

[@itsthesteve]

func func func func func func

[@vikingthedude]

3:24 what happens if we just call wg.Done() after calling process? Ie without defer?

[@IllllIIllllI]

The only difference is that defer will execute even if `process` panics. Without defer, it would possibly cause a deadlock

[@dreamsofcode]

defer is a little safer, especially if the logic changes later on. However, in this example, it'll also work fine.

[@vikingthedude]

I love how youtube decides i should see these replies only after 6 days

[@temie933]

Can you make a neovim for elixir tutorial? I keep getting stuck on it myself.

[@charliecandimaunten1635]

I don't know much about Go, so forgive this potentially stupid question: are these basically generators as defined in Python?

[@luanbui5273]

I think Go is only fun when you learn about the syntax. When I use it, I don't get it, it's like a primitive thing. Most Go developers won't update Go to a new version because they don't need it.

[@joaodiasconde]

Go lang and programmers slowly realizing have some more features may actually be good 😆 Watch it basically doing what other languages do but worse because they didnt accept complexity from the start and think "simplicity" is cutting corners

[@fsharplove]

Even "they didnt accept complexity from the start" is debatable. if err != nil all over the place.

[@sevos]

Ah yes, another one screaming "quit having fun"

[@jaymartinez311]

in the end it’ll have to add the other stuff anyway. So i like how it started but they need to rev-up some new declarative functionality. We are in the age of AI. This function should just work. I shouldn’t need to know the details of it unless i click through the different function signature docs in the ide.

[@ThomasWSmith-wm5xn]

a 4:18 iwas really hoping youd describe an async queue. nope - just some bs function iterator i don't need or want :(

[@mailman2097]

seq2.. why no method overloading?

[@jaymartinez311]

I need a slice keyword because i’m slow and confuse a lot of things 😂. This feature is great but i need something more declarative where im just calling a keyword & it just works! Yes im a bad developer but swiftui’s declarative nature has ruined me and made me lazy 😂. No more imperative button code. You just call button and it’s there. I need this in a package or more preferably in the language. Ok it started off minimal but when i first used rust is was basic, now you barely see lifetimes because they have slowly abstracted them into keywords where you can literally build your own state manager with miner lifetime usage to get it done. Pack the language with declarative stuff! rant over. 😂

[@antonbozhinov]

There is a difference between pretty to look at and easy to work with code. I think this feature of Go is neither

[@egor.okhterov]

This is not the go way. One of the key distinguishing features of go is its relentless simplicity. We don't need new syntactic sugar. What we need is more speed and less memory consumption.

[@OldKing11100]

For the life of me I couldn't get your iterators to work. I decided to go with a plain old function that wraps another function instead. func GoRange[T any](list []T, gofunc func(int, T) error) []error { var ( wg sync.WaitGroup mu sync.Mutex errs []error ) wg.Add(len(list)) for i, l := range list { go func() { defer wg.Done() if err := gofunc(i, l); err != nil { mu.Lock() errs = append(errs, err) mu.Unlock() } }() } wg.Wait() return errs } EDIT: changed to use mutex for performance

[@tauiin]

Personally, I prefer generators in the style of python for something like this (e.g. the nicety of just using "yield" instead of this strange function stuff), but I guess this'll do. I think the best part that will come out of this is watching all the gophers complaining for years about how this too complex... "What's next???? Usable Enums?!?!? Preposterous I say"

[@TopSwagCode]

Looks a lot like what is done in C# / Dotnet. I like the comment you mention, it's hard to tell how the underlying code is run. I would say good naming could help with this. But for love of god, please don't do like dotnet, by making everything a Task and have async / await all the way up the stack.

[@Saturate0806]

this is huge

[@memory_leaps]

MANN I WAS JUS ABOUT TO COMMENT IF U COULD MAKE MORE GO ADVACNED CONTENT SOO GOOOD

[@earnstein7607]

Great but I'll stick to the first implementation.

[@CoolestPossibleName]

Still waiting for arena in go

[@siya.abc123]

Did they hire a JavaScript programmers in the core team?

[@idiomaxiom]

that's a lot of steps to run a par...

[@trejohnson7677]

need this shit.

[@botyironcastle]

just use c# and linq and parallel MUCH MUCH more reasonable

[@unpin8101]

ah yes, features that exist in other languages for years

[@pluieuwu]

oh... they finally discovered operations on iterators...?

[@BarakaAndrew]

This is great, gonna make some things way easier to do

[@mr_wormhole]

i have a bad feeling about this

[@RishabhGupta-tn6do]

Fall of golang, sadge.

[@paw565]

Nice video, but I'm too stupid to understand it.

[@maleldil1]

While I've always been a critic of Go for being anemic in features, this one feels out of touch with the rest of the language. It shies away from Go's main idea of simplicity.

[@irlshrek]

I hope you start making Rust content

[@Robert-G]

this seems to be the worst implementation of c# iterators (or js generators) I’ve seen so far. by a lot. This must be the most convoluted and arcane syntax to solve the problem in any language. I would have expected the func to return an interface that for .. range can iterate over, and you’d yield a value in your func to the consumer. the compiler would build a hidden struct that implements the interface. (haven’t written go in a while, last time, they didn’t even have a standard interface for something that you can loop over) that way you can process a stream in your func and consume each item in a for loop the moment they’re read without any expensive extra threads or synchronization (when using go routines and channels) go is such a bizarre language…

[@ThomasWSmith-wm5xn]

hate; js flashbacks

[@billgrant7262]

WE DON'T WANT TO CHANGE THE WAY TO WRITE LOOPS! THAT'S THE ENTIRE POINT OF GO!

[@sunofabeach9424]

wow, Go has finally got yet another feature that exists in other languages for like forever

[@pristavucalin9338]

Govascript 😅

[@element12144]

Congrats on making Go look just as shitty and unreadable as Scala. Generics was such a mistake, dear lord.

[@everyhandletaken]

It's all too complicated 😢

[@funky_hedgehog]

Rust will be easier to use than this.

[@korseg1990]

the syntax is terrible

[@DennisWethmar]

I dont like it

[@adicide9070]

uu can this change it so that we don't write fucking loops no more? :D

[@melodyogonna]

They could have simply added generator rather than this ugly shit. A simple yield statement is simpler and far more readable

[@svetlinzarev3453]

There is no point in using Golang, when there is Rust. Go is just worse at everything

[@dreamsofcode]

I like Rust, but I think they both have their own strengths and weaknesses. There's no equivalent yet for Go+Templ+HTMX when it comes to Rust. Additionally, Concurrency in Go is better than async in Rust imho.

[@iatheman]

@@dreamsofcode Agreed. Async and lifetimes are great ideas, but in practice they make everything exponentially more complex and boring to read and write as a dev. Rust did simplify lifetimes as much as they could, though... maybe it's the same with async, but the end result is still far from enjoyable or productively readable.

[@baxiry.]

May the Almighty God bless your steadfast faith in Holy Rust ✝🛐

[@apestogetherstrong341]

I hate golang, for very deep reasons. From the fact that it’s shitty by design, and the fact they cover it up with “simplicity” (see clojure for REAL simplicity, and not just easiness), all the way to the fact that this language forces programmers to be primitive code monkeys that cant do anything besides write lines of unexpressive gibberish. verbosity language has the repetitiveness and verboseness of assembly, but without the benefits that verbosity gives to other languages. It has the shortsighted sloppy design of C, but without the ability to do what C is capable of. I hope they keep piling features on top of Go so that all these new features will eventually reveal more and more of the fundamental problems with the language, just like once happened with C++. Trying to evolve C is a bad idea, it’s proven by all the garbage latguages we have today. The only ones that are designed relatively well are Erlang, Clojure and Common Lisp.

[@neonraytracer8846]

Calm it down.. I think the languages you mentioned as having good design, has horrible syntax and is tough to read. We all have different opinions, but nobody agrees that C is sloppy or shortsighted. That's simply not correct especially given the history of things. Take a chill pill mate

[@iatheman]

What a salty way to praise your personal taste.

[@baxiry.]

The beauty of Haskell, Erlang, Clojure... comes from the fact that they tolerate the problem of copying data everywhere. And eat up as much memory as possible. If these languages try to solve this problem...then they will fall into what you call: bad design.

[@apestogetherstrong341]

@@baxiry. Clojure solved this problem without falling into bad design. Please don't show your lack of knowledge. Clojure data structures don't copy data everywhere thanks to what they call „persistent data structures”. Erlang employs this technique too, where it's important. I never mentioned Haskell, you did. I don't like Haskell. Do your research before talking. Study efficient persistent data structures, hash-array mapped trie, rbb trees and others. There even are implementations in C++ (see „immer” library), where these exact data structures allow for massively efficient code (See the talk „Postmodern immutable data structures”). What you described is false and your opinion is based on misinformation. Have a good day.