For loops are redundant

Any Python for loop can be written without using "for".
Seeing how to do this rewrite is the key to understanding how iteration fundamentally works in Python.
Note: PLEASE, DO NOT REPLACE YOUR FOR LOOPS. This video is strictly for understanding what happens under the hood with iter/next/StopIteration stuff under the hood in CPython.
Errata:
- Not all for loops raise a StopIteration, only the ones that stop do, i.e. not infinite loops.
― mCoding with James Murphy (mcoding.io)
Source code: github.com/mCodingLLC/VideosS...
SUPPORT ME ⭐
---------------------------------------------------
Patreon: / mcoding
Paypal: www.paypal.com/donate/?hosted...
Other donations: mcoding.io/donate
Top patrons and donors: Jameson, Laura M, Vahnekie, Dragos C, Matt R, Casey G, Johan A, John Martin, Jason F, Mutual Information, Neel R
BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------
Discord: / discord
Github: github.com/mCodingLLC/
Reddit: / mcoding
Facebook: / james.mcoding
MUSIC
---------------------------------------------------
Local Elevator by Kevin MacLeod is licensed under a Creative Commons Attribution 4.0 license. creativecommons.org/licenses/...
Source: incompetech.com/music/royalty-...
Artist: incompetech.com/

[@189Blake]

I'll replace all my for loops with while-try statements! I bet my co-workers will praise my code on the code review!

[@mCoding]

Don't even joke about this!! I take no responsibility.

[@necroowl3953]

@@mCoding then don't fucking make a video trying to reinvent the wheel dumbass. What do you expect? Only praises from Indian kids trying to learn how to hack android phones? _"elo frens"_ grade of video, bro.

[@hikari1690]

Yeah, the key scratches on your car are them signing it out of admiration...

[@drygordspellweaver8761]

Make sure to put this in the comments: ( * Y * )

[@InvestLarge]

@@drygordspellweaver8761 You're the best, or as the young ones say now "the GOAT"!

[@danielgysi5729]

All loops are redundant, just use goto. Better yet, write everything in assembly and use jmp

[@MithicSpirit]

you need at least conditional jumps for asm to be turing complete iirc

[@viliamklein]

Since hard drives are being phased out, my magnetized needle is no longer a good programming tool. What can I use to flip gates in flash?

[@bereck7735]

@@MithicSpirit hello mithic, nice to see you

[@DeShark88]

@@viliamklein Ionising radiation particles generated from your synchrotron (or cyclotron if you can't afford a synchrotron). Of course, emacs has a command for generating synchrotron radiation from your terminal...

[@mCoding]

This actually reminds me of a great talk about the movfuscator. Believe it or not, x86's MOV is all you need! So yeah, loops are redundant, but it's kind of a good thing. Highly recommend watching the movfuscator talk.

[@georgplaz]

I hope nobody sees this as an invitation to not use for-loops. I had discussions on SO about this very topic ._.

[@MustbeTheBassest]

Yeah, as a newbie I don't understand the point of the video and especially the title. It sounds like for loops are bad. But in the video, the op seems to be saying, "yeah, it's dumb, but better than doing something else." Is that the right takeaway?

[@fppt1555]

@@MustbeTheBassest No, you can read the description if you didnt understand his point: In this video he's teaching how iteration works in Python, behind the for loop abstraction

[@georgplaz]

@@MustbeTheBassest The video just says "this is what happens behind the scenes". use for loops, they are great :)

[@deimuader]

@Adrian Claudiu Covaci but it's recommended not to use these functions

[@d00dEEE]

@Adrian Claudiu Covaci But even when learning about those mapping functions, you'll need a grasp of how iterators are implemented, which is why James produced this video. He's showing "here's what's going" from sort of an infrastructure perspective, so now you can move on to writing classes with __iter__/__next__, generator functions and map-like functions that consume all of those things.

[@nicholaskoldys1450]

FOR loops execute code for a set specific condition set at onset, for a KNOWN interval. WHILE loops execute code for a set specific condition for an UNKNOWN interval. Code should be designed under these principles as they lead to greater readability.

[@MaxMustermann-bm7qt]

Jup, while loops can therefore not be parallel processed.

[@siriusradheoff8361]

Yes but why not make for loops be able to do that too? I have always been annoyed that you can't turn everything in a for loop into a variable. for i in range(x, y, z):{something that can actually change x, y, and z as they goes on} Python's redundancy is an unnecessary distraction.

[@MaxMustermann-bm7qt]

@@siriusradheoff8361 congratiulations, you just slowed down every for loop in existence to avoid using a while loop. - You want a new function based on while. - As i assume demand for that is limited i propose writing it yourself, don't try using for it cannot work. - Also x needs to be known beforehand.

[@siriusradheoff8361]

@@MaxMustermann-bm7qt But why should things be that way? Python got rid of do-while. Why not get rid of while as well? For loops will do everything. Simple for loops will run very fast because they execute the C for loop. More complicated ones unpack the power of abstraction. Also you only need the initial value of x. Later x itself can change. Good for searches where after searching in a range for a while, a different range suggests itself and you can run that. A single for loop does everything

[@MaxMustermann-bm7qt]

@@siriusradheoff8361 yes, that means it makes no sense setting it inside the function. It is only used once, at the start. It doesn't change. It is never used again and only an Initial condition. Forgetting it makes the code fail. Any you are like yeah pass in any variable, but define it in the first line of the function or the code breaks. Just pass it in as a numerical argument. All info to set an IC must be known beforehand! Always. No use setting it in the loop. I could build an airplane out of my couch, but that is just pointless. I could build hidden wings in every couch i own, so people can decide wether they want to sit or fly. The point is usually people know wether they want to sit, or fly in their livingroom in advance and get a plane or a couch for the job. What you are saying is: Augment "for" so that it can replace while and do so. Instead of: replace "for" as one could always use "while" to do so without changing it. - the only reason for exists is speed and convinience. - the augmented "for" is pretty inconvinient for most while loops, if at all possible. While is: Check cond. ->do stuff check cond, do stuff.... For is: set increment to IC, do stuff, increment, check limit of increment-->loop In a for loop "do stuff" doesn't change the increment, if it does because you changed the functionality that is a pretty big deal to get used to.

[@lamprospitsillou6325]

I was expecting you to dive in to list comprehensions, for_each methods ,map methods, filter methods etc.

[@mCoding]

While the for loop is indeed equivalent to many of those other things, in Python (CPython) it just so happens that the semantics of a for loop are defined in terms of while, iter, next, and StopIteration, not in terms of those other things. This video is not meant to tell you to replace your for loops (as I mention in the video DON'T actually do this), but rather to show you what for loops are built out of.

[@lamprospitsillou6325]

@@mCoding Yeah, from what i understood the video was more about HOW loops work in python, not specifically about the for loop and it's replacement (which was never stated in the video, a bit in the title but it is what it is 😁). Maybe do a follow-up video on those methods,it would make for an interesting video ! Keep up the good work, you make some unique videos!

[@johnnytoobad7785]

This is classic "why simplicate when you can complify". The GEEKS creed.

[@AbhinavKulshreshtha]

True.. But it is also about learning under the hood stuff.

[@TheDoomerBlox]

Huh? It's an explanation of the internal logic, should you wish to reimplement it in a language that does not provide it for you. That, as well as a demonstration of how odd it would look reimplemented inside Python itself. don't seem very make-it-complicated to me (though the title IS geek clickbait)

[@mCoding]

Indeed, please DO NOT actually replace your for loops with their while equivalents. For exists for a reason, because the concept of looping over something is so incredibly common, introducing this pattern and giving it its own syntax is almost necessary, just as if you wrote the same code 10 times you would benefit from creating a function and calling that instead. This is more for understanding what the for loop actually does under the hood instead of thinking about it as a black box, because these details about StopIteration and calls to iter/next do come up from time to time.

[@SufferDYT]

@@mCoding Then why did you title the video in such a way that you would expect to hear the argument for why you shouldn't be using for loops?

[@omari6108]

Thanks a lot for this video. As someone familiar with Python, but still have pending projects that need completion, I enjoy the clarity of your videos in how you explain concepts. Anytime I see more insight into what’s going on behind the scenes, it does help me understand what’s going on much better. Practical info is one thing, but useful info will always be useful.

[@mCoding]

Thanks for your kind words! I appreciate it!

[@jithin.johnson]

Try @anthonywritescode

[@pradyumnakatageri3475]

Hi James. Can you explain generators with examples on how to handle them for file parsing like logs as it is a good practical application for generators.

[@thestemgamer3346]

Generators evaluate lazily. This essentially means that every value ia created when the generator needs it, rather than held in memory all at once. This allows you to form infinitely long generators. The yield keyword simply represents the next value you want to generate, and every time you call the generator, yield will toss out the next value you want to use. The main use of generators in parsing is for constructing parser combinators which will parse a file byte by byte without requiring you to store the entire file in memory. Allowing you to parse potentially massive files with constant (not exactly) memory.

[@Amipotsophspond]

for loops under a 1000 iterations can be replaced by recursion. over 1000 you will need to split the loop in to chunks.

[@sylascoker1483]

Why specifically 1000?

[@richardlange7598]

@@sylascoker1483 probably some sort of limit on recursion to prevent stack issues .

[@palmberry5576]

@@richardlange7598 mmm yes, I love it when my loops cause stack issues

[@gunahawk6893]

recursion takes extra space , always use iterations

[@donepearce]

Great set of reasons to just use a FOR loop.

[@mCoding]

Indeed! Although understanding the internals can be useful, we should all still just use for loops.

[@SufferDYT]

@@mCoding This video isn't useful.

[@MrDoomedtofail]

@@SufferDYT it's useful for a better understanding of what python is doing at a lower level, even if it isn't exactly practical

[@kkjr6673]

Nice gentle introduction to the concept of Turing equivalence. I like that you point out that readability and efficiency in the chosen language/compiler are the relevant reasons for picking one construct over another.

[@donsheehy]

The last example is an especially useful pattern if you need to do something special with the first item from the iterator.

[@sebastianjost]

I've had about 10 introductions to python and I think every one included a task like "solve this once with a for loop and once with a while loop and once without a loop." So this was always obvious to me. You don't even need for or while. You can use recursion instead. Obviously you shouldn't do that, but it's a nice option to keep in mind.

[@Milecarful]

This is not applicable in practice. Recursions cannot handle infinite iteration, loops can (with lazy evaluation). The weakest link in recursion is its non-constant memory overhead. Iteration doesn't necessarily need anything over a constant memory overhead. So in an absence of an infinitely large stack, you do, in fact, need at least a jump statement, even if the task is just infinitely writing out a constant character.

[@orange1304]

@@Milecarful As a side note, in some other languages (mostly the ones following functional paradigm, like lisp dialects), infinite recursion is possible through tail call optimization, for anyone wondering. The concept doesn't exist in python however, so no infinite recursion there, sadly :-(

[@Milecarful]

@@orange1304 But tail call recursion optimization doesn't enable you to transform all infinite iterations to recursions, just the simple ones. Recursions in a general case do not have any guarantees on time and space complexity, just like loops.

[@AshtonSnapp]

Similarly, for loops in Rust take an object that implements the Iterator trait (or the IntoIterator trait, which returns an Iterator implementor) and calls its next() method. This method returns an Option, and the loop ends when that return value is a None instead of a Some.

[@cerulity32k]

Rust can do what basically the range() function can do, with (10..20).step_by(5), which I haven’t yet seen in other languages, let alone one that is so low level.

[@altered_dev]

@@cerulity32k kotlin has that too: 10..20 step 5

[@theairaccumulator7144]

@@cerulity32k you haven't seen it in other low level languages because going through 5 hoops to call a function every loop iteration is pretty stupid and inefficient. Maybe Rust optimizes it but it would still be slow.

[@jaysonbunnell8097]

A monad is a monoid in the category of endofunctors...

[@diegoaugusto1561]

@@theairaccumulator7144 c++ iterators are just a pointer to the start and another to the end. pretty dumb and inflexible if you ask me

[@williamplayle-devries5682]

I love watching these because even though I think I know the content, there's always something new to me!

[@GanerRL]

why is it that in a list comp you can get the itterator variable in locals() but not in a normal for loop?

[@mCoding]

Oh my! Well I guess this is somewhere between implementation detail and bug that would be very hard to fix. The variable name is ".0" which is not a valid name. Probably just a limitation of how comprehensions are implemented. Fun fact though, you *technically* can access the iterator in a normal for loop if it is tracked by the garbage collector. expr = [1,2,3] for x in expr: y = gc.get_referrers(expr)[0] break print(y)

[@GanerRL]

@@mCoding oo thats clever lol

[@aleksandersabak]

Have you considered replacing your while loops with recursive calls and exceptions with `Either` type values? This rids of almost as much redundancy as possible.

[@XCanG]

What? Simplest example what you mean?

[@aleksandersabak]

@@XCanG what was said in the video is that a for loop can be expressed as a while loop with some error handling. In principle every loop can be expressed as a recursive call and all error handling can be done by returning error values, so the for loop turns into: def for_loop(iter, body): match next(iter): case StopIteration(): return case val: body(val) for_loop(iter, body) Which almost works. The problem in python is that errors aren't return values so they have to be caught. If the iterator started returning StopIteration at its end instead of throwing it my code would work, and it's similar to how functional languages handle loops and errors.

[@XCanG]

@@aleksandersabak Ah, I see. I mostly work with Python and JS and that behavior was normal to me. Didn't think about that. Also I still didn't move to Python 3.10 due to work constraint on previous version, so I didn't use new features like match-case.

[@aleksandersabak]

@@XCanG Structural pattern matching is the standard branching construct in functional languages. I'm very glad it's made its way into python.

[@dalmationblack]

I went through basically this exact lesson except in Java for one of my CS classes, so I was sitting and nodding along right up until you said that the loop exits by raising an exception... WHO DECIDED THAT WAS THE WAY TO DO IT??

[@JansthcirlU]

In C#, the equivalent iterator approach returns a Boolean value instead of throwing an exception, very surprised they didn't use the same approach in Python

[@pardhabandaru1105]

This is expected as per Zen of Python "Easier to ask for forgiveness than permission"

[@Omikronik]

You can also replace a for/foreach with LINQ most of the time

[@esbensloth]

​@@pardhabandaru1105 Except in this example we are asking both for permission and forgiveness. Even though the permission conditional is constant it is there.

[@Milecarful]

@@esbensloth You misunderstand. This example breaks the Zen of Python. That's why you should never use this. But the way next is built doesn't. You're just not supposed to use it like this.

[@user-ni2we7kl1j]

@@Milecarful Still, the implementation looks really cursed to pretty much any programmer. Using exceptions for anything but failure handling is generally considered an antipattern

[@leostantontowne197]

I really enjoy your videos and have learned a lot from you. Yours are the 'deepest" Python videos on KZbin IMO. I think it would be great if you would expand to cover Clojure. There are many Clojure talks online but almost no 'how to do it" or "how it works under the hood" Clojure videos. I also think you would enjoy Clojure.

[@twistedsim]

In Python 3.9, you could do something like this: data = [1,2,3] sentinel = object() it = iter(data) while (d := next(it, sentinel)) is not sentinel: print(d)

[@mCoding]

Neat! Does it still work if you just replace sentinel with StopIteration?

[@twistedsim]

@@mCoding The exception is not raise, so it would still need a check if the default value is equal to StopIteration (`while (d := next(it, StopIteration)) is not StopIteration:`). That would also prevent a rare edge case where the data actually contains `StopIteration`.

[@kitgzz]

fantastic. so for loops are redundant by using while loop. What an awesome instructor!

[@CakeBossan]

did you completely miss the last minute? it is a useless video, but i guess it teaches StopIteration

[@johngrear6506]

Yeah, it's a click-bait title and I fell for it.

[@mCoding]

@CakeBossan Indeed, the secret point of the video is to teach you how StopIteration underpins iteration in Python, so I guess I failed successfully. StopIterations are rarely every used explicitly, and in that sense it is "useless" but I think it is very important to understand they are there.

[@ssholum]

@@mCoding It doesn't take much to "learn to code", but it does take quite a bit to "learn about code" (and there are far fewer friendly resources on the topic). I appreciate your "useless" video topics like this one.

[@Mutual_Information]

lol using an exception like that feels so dirty.. but maybe there is a broader design principle at play that just never got wide adoption? Something like.. exceptions aren't just to signal broken code - but to signal any deviation from operations that yield the typical return type? Idk, still feels dirty..

[@MithicSpirit]

Yeah, the way Rust handles error-prone code feels much more natural for an application like this. Basically it uses an "object" with two distinct (and disjoint) constructors where one represents the normal state ("Ok" or "Some") and one represents the error state ("Err" or "None"); then you can do pattern matching to do different things based on the constructor used.

[@georgplaz]

using exceptions like that is actually part of the "EAFP" design philosophy of python. You should actually access a dict and catch the KeyError, rather than check if the value is contained peforehand. It might feel dirty coming from other languages, but in python it is perfectly OK to use exceptions for code flow control

[@DeShark88]

@@georgplaz European Association of Fish Pathologists? It definitely smells fishy!

[@mCoding]

As long as everything is consistent I won't object. There were design challenges that led it to be this way, although it does seem fishy coming from other languages. In the Python world that's just the way it is. Would be interesting to imagine a world where Python returned error codes instead.

[@masheroz]

@@georgplaz why not dictvar.get(key, None)?

[@dabusdruva]

Hi James. Fantastic work. These videos are great. I was just watching a presentation by Mark Shannon, principle engineer on the CPython team. He was talking about stack frame optimizations they have made in 3.11 for speed improvements, and one of the points that he touched on is that 3.11 generates a stack frame lazily when it needs to, like when an exception is thrown. His reasoning was sound, optimize for the most common case, even if the non-common case might be a bit slower than before. As such, regular execution is faster, but raising exceptions is slightly slower. But as we've seen in this video and previous videos you've made, python expects StopIteration exceptions in the normal case. Couldn't this have unforeseen consequences on code that relies very heavily on iteration? I'd love to hear your thoughts on this. Thank you very much for taking the time to read my comment.

[@mCoding]

Thank you and great question! In theory this could have a minor performance impact for iteration heavy code, but they do benchmark a wide variety of code to weigh the benefits. I suppose only time will tell, but this is but one of many optimizations being made and overall 3.11 and on continue to be faster and faster so it is unlikely that remaining on 3.10 will be a long term strategy for speed even in iteration heavy code.

[@Spedley_2142]

I was doing a lot of assembly years ago (picmicro) and made my own macro loop. do ... while ... loop where the main do loop executes and the while statement acts as a break. Missing out either of the [...] statements turned it into either a normal do loop or a while loop but allowed for inittialisation within it, e.g. do chr=next_chr() While chr!=null Print chr Loop

[@Bruh-sp2bj]

Print() is redundant, lets use sys.stdout and flush

[@mCoding]

Well, you're right that it is redundant. That doesn't mean you shouldn't use it!

[@AByteofCode]

I've been trying to figure out ways to make as many python keywords as possible redundant, and I'm currently left with def, while (recursion limits :/), await, async, try, except, finally, raise and nonlocal. Would anyone have any ideas on how any of these could be replaced with other features?

[@mCoding]

Hmmmm... you can definitely get rid of await and async since those are just wrappers around generators in Python (for now). Nonlocal might be avoidable using unique variable names. Theoretically try except finally raise could be avoided by rewriting everything to never throw exceptions but beyond that they would be very hard to avoid. I think you could get rid of except by always using try: finally and inspecting whether the current exception is set or not, but not totally sure on that one and it would involve accessing sys, which is not on your list. I think you might need to add exec as well, or were you able to rule that one out?

[@SkyyySi]

Now I'm worried about your next video...

[@randomidiot13_]

I'm assuming they mean the keywords per the keyword module, in which case accessing sys can be done with dunder import and exec is fine as it is a function. I'm not entirely sure you can get future imports without from and import.

[@AByteofCode]

​@@mCoding Thank you very much! This is a great help for my video about it :) I'll give you credit for these. Also found a way to simulate raise, by calling .throw on a generator. The traceback is a bit wonky but everything is for this challenge :)

[@QuantumHistorian]

Can you not get rid of def by defining lambda's and assigning them a name with =?

[@timecubed]

The iterative part of for loops are actually done in c making for loops pretty quick apart from any code actually inside the for loop. While loops have their iterative part coded completely in python, meaning that while loops are actually way slower than for loops. Also, both are extremely easy to read, but I would consider for loops more readable than while loops when iterating over an array. It's either 'for each item in array' or 'while true, take the next item in the array'. Also also you would have to write a check at the very end of your while loop to check if you've reached the end of the array which makes them even slower for this task.

[@joseville]

4:00 behind the scenes, does python call iter(it) as well? And does that iter(it) return `it`, the same iterator or a new one?

[@nnirr1]

Might be wrong on this one but i think it does call iter everytime and the iterators themselves are implemented to return self

[@mCoding]

Yes that's correct, Python does call iter and next under the hood, so if you define a dunder iter it will be called, and you are also correct in that iter(it) is it. Iterators are required to also define a dunder iter that returns self, so it returns the exact same iterator.

[@dedpossum66]

@mcoding You’re missing an important example IMO. There is a way to avoid the exception entirely. You can provide a second value to ‘next’ which will be returned after your iterative has been expended. In particular, if you provide ‘None’ as the aforementioned argument in the case that the iterator will not return that result, then you can use the condition ‘ is not None’ and avoid the ‘StopIteration’ exception altogether. Of course, the condition that the iterator will not return null is essential for this to work - so perhaps it is a bad example?

[@mixedbytc]

I'd argue that verbosity is more redundant when there's already more readable syntactic sugar for your intentions. Still, it's good that you point out Python's exception-based paradigms. Rust follows a similar strategy with 'for/in', but with error-passing.

[@soupnoodles]

1:48 Is the bug that it need not always throw an exception? For example with a itertools.cycle object otherwise my best guess is that there could be a problem with that variable assignment there edit: ah i was wrong

[@mCoding]

Good guess and actually you mention another good point, for loops don't *always* throw an exception, since some of them are infinite loops!

[@lucasvsl889]

Why would you add the else with the exception? The break statement already ensures that subsequent code won’t be executed if a stop iteration is raised. I think decreasing the indent seem cleaner… 🤔

[@mCoding]

As is shown in the video, the else is put there to express intent, not to be cleaner or shorter. The else after a try except unconditionally means what to do when no exception is raised, which is easy for a reader to understand. This code isn't particularly complex, so the other way is perfectly fine as well, it's just up to preference.

[@ethandavis7310]

The one thing I hate about python for loops is that they're not really for loops in the classical sense, they're foreach loops on an array of sequential integers. This is fine for almost all applications, except for if you want to perform arithmetic on your index inside the loop nothing happens, which is REALLY annoying to debug if you forget this fact. Basically if you do i - - inside the loop you're decreasing the ith entry of this array, but on the next iteration you're accessing the i+1 element which has not has the arithmetic performed on it

[@anselmschueler]

Does the method of getting the iterator early actually mutate the iterator within the for loop? Does calling next(it) in the for loop skip an iteration?

[@MattSpaul]

Only using while was the first thing I had to get used to when learning Fortran and I've stuck to it ever since

[@MaxG628]

“Understanding iteration is the key to mastering any programming language.” *Cries in Haskell* Glibness aside, understanding how to do something repeatedly, whether through iteration, recursion, or an abstraction over those like map and fold aka reduce, is super important.

[@mCoding]

I didn't mean to exclude haskell! Coming from a math background I consider recursion to be the ultimate form of iteration.

[@illegalsmirf]

The key to Python is to continually reinvent the wheel in order to make CS grads look smart

[@QuantumHistorian]

As a question to the people who code regularly, what's the relative frequencies of *for* and *while* loops in your code? I find I very rarely use while loops, only in specialised contexts like I/O that need to happen endlessly, while for loops are basically everywhere. This is true whether I'm doing python, C#, C++, or C

[@dIancaster]

Same here. For loops are a mainstay, maybe one in every two functions, while “while” loops are once per project, if that.

[@drygordspellweaver8761]

For me it’s 30% while loops but only because I do a lot of variable duration waiting for certain conditions to arise. while getState() is None: Time.sleep(1)

[@ABaumstumpf]

c++, mostly range-for "for( const auto& item : collection)" cause most of the time (> 50% i would guess) i am iterating over every element and want to apply some logic. A "normal" for( int i = 0; i < collection.size(); i++ ) when i need the index inside - like when checking each element against all following element. We have a few programs where we have 2 related same-sized containers and need to check if all elements after the current one fulfil criteria based on the current element of both containers - aka conditions like "element A[i+j] must be smaller than A[i] and have at least one of the properties from B[i]" and while-loops when i either need to loop over multiple containers at once, when i loop over the container in multiple steps, when not looping over the entire thing, and specially when it is not known how long to loop. So things like IO and logic-loops are often a "while(true){runLogic}" or "while(processEvents){}".

[@etopowertwon]

Reminds me that Rust internally (at MIR stage) transforms for loops into endless loop{} from which it `breaks` once iterator is exhausted.

[@alphanow4199]

no! whilz are redondant, because you can write for _ in iter(lambda: bool(, False)):

[@mCoding]

I was going to give this as an exercise! I think these being able to do these things shows a great deal of knowledge of the language (even though we obviously shouldn't replace fors or whiles with the other). So great job!

[@tjmoody5932]

x = 0 while x < len(expr): print(expr[x]) x += 1

[@piotrmazgaj]

Did I ever mention you Roko's Basilisk thought experiment...

[@mrKreuzfeld]

By the same note, whileloops are redundant in many languages, one can just use goto and label :D

[@yogeshchandrasekharuni4054]

hey @mcoding, is there any complete cpp tutorial series on your list? even if you publish it on Udemy, i bet there's a bunch of people like me who need it - cheers

[@kashgarinn]

I encountered a bit of weirdness with pytest where the yield operation was used with fixtures as a way to clean up after the fixture was used. I wonder if yield and iterators could be used in other ways that are neat but not obvious.

[@PanduPoluan]

That's actually pytest hijacking Python plumbing to implement things in a (in the makers' mind) simpler, more readable way, with minimal boilerplate. So it's neat but totally not obvious as internally pytest will need to run the resulting generator to get the dependency to inject, and then don't forget to continue the generator to execute the 'cleanup' part. This is not the only Python plumbing that pytest hijacked; pytest also hijacked the "assert" mechanism to be able to provide a detailed result when a test fails.

[@waya208]

Another alternative (no for or while loop required): expr = [1, 2, 3] print(" ".join(map(str, expr))) 🙂(just for fun)

[@fallingintime]

I guess you can do while X := next(it): and pass a sentinel value None to the iter function to avoid exception handling. But you would have to ensure there are no falsy values in the list

[@pascalfragnoud2846]

You could also use while x := next(it, False). Doesn't solve the problem of false values though.

[@user-pl6hc4kj1o]

Is this "redundancy" behavior also the case in other languages, like C/C++, js, and java, or only python?

[@mCoding]

Yes this relationship is common in most programming languages. A for loop can always be implemented in terms of a while loop. In most languages a while loop can also be implemented in terms of a for loop.

[@sinakarimi8273]

But as i read about while loop, it is very slow in python, what is the advantage of this technique over for loop?

[@Milecarful]

Quite literally nothing.

[@mikopiko]

I wish there was a similar channel like you but dedicated for Ruby

[@ChristopherRucinski]

`it = iter(expr)` could be assigned with the walrus operator on the first line of the for-loop, no?

[@ABaumstumpf]

As with many things in python - brilliant and horrendous at the same time, depending on how you look at it. On thing for sure - it is insanely bad in terms of performance specially for small loops, but then again that is python. With the for-loop it seems mostly insane to throw an exception cause it is not actually an exceptional occurrence for collection to end but rather the opposite - it is the expected behaviour. You specifically write a loop to go through all elements of a collections so you know it must end at some point. the "next" function even gives you the option to use a sentinel which would make a lot more sense here. On the other hand it is great that next can throw an exception when going beyond the last element - useful when you your self want to iterate over some elements, and very much needed when the end-condition is very complex or the underlying data might just "vanish" while you are still working on it.

[@everytimeifall]

We could create an infinite generator with recursion, and use for loop instead of the while statements. Rendering the while a redundant. Ignoring the recursion limit ofc

[@user-on5ps4ik6e]

Hi, could you explain how for works with __getitem__ and __iter__, how for works if we define only getitem?

[@nnirr1]

If iter method is implemented, it will take priority over a potential getitem. If only getitem exists, it will be called with the key 0 and increasing with each iteration until a StopIteration exception is raised from the method.

[@mCoding]

Indeed this is a tricky quirk. iter() does not call dunder iter. It does if dunder iter exists, but if dunder iter does not exist and dunder getitem does exist, then it creates a special kind of sequence iterator under the hood whose dunder next method calls the original object's getitem with 0, 1, 2, ... The only thing I have to add beyond what @Nir mentioned is that it is customary for a dunder getitem to raise an IndexError instead of a StopIteration, since it is possible people call your getitem directly using my_object[some_big_index] and they would be expecting an IndexError in that situation. I just tested and both IndexError and StopIteration work as expected in the case of iteration with getitem.

[@nnirr1]

@@mCoding Thank you for the correction ☺️

[@jackgerberuae]

What is the reason behind avoiding for loops. Speed, or just a bee in someone’s bonnet?

[@mastermati773]

Whiles are redundant too. You can use if with recursion.

[@mixedbytc]

Not sure about Python, but might that lead to hellish branching and jumping in systems languages?

[@mastermati773]

@@mixedbytc To be very honest, it would be even worse. One could simply run out of space on stack. The point I'm trying to make is that in theory you can dump all loops, but you miss the purpose and coming with it optimizations (e.g. vectorizations).

[@kvelez]

1:23 def main(): expr = [1,2,3] it = iter(expr) while True: try: x = next(it) print(x) except StopIteration: break # else: # raise StopIteration if __name__ == "__main__": main() 3:05 Explanation.

[@jonathanmoore5619]

Thks James

[@alexandarjelenic2880]

thanks for the intro, and warning

[@Jelissei]

Is the for-loop just a while-loop in disguise? As in while-loop is what is actually happening, but the for-loop is shorter, thus more practical? And two teachers told me, that using breaks is bad coding. Any opinion about that?

[@masheroz]

Breaks are great for short circuiting. If you're looping through a list to see if a value exists, why do you need to check every element if you've already found it? Break out of the loop. A for loop is an alternative syntax for a specific type of while loop.

[@Jelissei]

@@masheroz thanks! That was my reasoning also, but they said there are usually better ways to achieve that. I'm not yet ready to know exactly what they mean though

[@Tsunazoz]

I thought the bug was that it doesn't check if "it" is iterable or not.

[@mCoding]

Good guess! But you can attempt to pass something that's not iterable to a real for loop and the iter call under the hood will raise a TypeError!

[@drygordspellweaver8761]

Good to know I’m not the only person who had that specific thought about the Robot Uprising and programmers... lol

[@osolomons]

Hello, I was surprised to find out that in python does not support "yield from" inside async generators - according to PEP 525, implementing it "would require a serious redesign of the generators implementation". I think this would make an interesting video if you're interested!

[@mCoding]

Thanks for this comment! I actually have never tried to yield from in an async generator before. It just never came up until now! I read that section of the pep and yeah it looks like the core devs may have backed themselves into a corner on that one, sometimes making the core parts of the language simple and functional makes it difficult to support these less common uses. It's certainly possible to make it happen but it would take a huge amount of effort by a very knowledgeable volunteer to implement it, so unless a demanding use case arises it may not happen this decade. Great idea for a video but i've proven myself not knowledgeable on the topic already, so I've got some learning to do first.

[@iyannazarian866]

You gotta love the brutal honesty here from both OP and Mcoding! Would love have a video explaining why that would require so much work.

[@smallface_]

Python does support yield, but I’m not completely sure how to implement it in this situation since I haven’t used it much

[@atrus3823]

I like the idea of using exceptions for behaviours of functions that are exceptions to the purpose of the function. It always seems really hacky and error-prone to me when a function returns special values under special conditions. That's what next would need to do in this case if it didn't raise an exception. It would need to return something that's not an element of the iterator to signal that it was done. This would kind of break the purpose of the next function, i.e., getting the next item of an iterator, since a done indicator is not an item in the iterator (or if it was this would add even more confusion). I think of it kind of like an IndexError. my_list[i] gets you the element of my_list at index i. If there is no element at index i, should it return a special value that is not a value in the list or raise an exception? I suppose another alternative is to always return 2 values (this is the Go approach) and have one of them be a done state, but this becomes really annoying in certain contexts.

[@ABaumstumpf]

"It would need to return something that's not an element of the iterator to signal that it was done." But next does NOT return an element of an iterator but, when existing, an iterator pointing towards the next element. And iterator it self is by definition not an element of the collection. And "I like the idea of using exceptions for behaviours of functions that are exceptions to the purpose of the function." Yes - so then why are you in favour of doing the exact opposite? Iterators are there to iterate over collections - their very purpose is to have a last element and stop after that. That is not the exception but their purpose.

[@atrus3823]

@@ABaumstumpf no, next doesn't return an iterator. It returns the next item. It advances the iterator in place. From the docs: "Retrieve the next item from the iterator." So the behaviour is an exception.

[@hikaritsumi2123]

I have done use while-loop in place of the for-loop in my early day of python coding simply because I don't know how to use for-loop and using something dumb like grab a length of the list, declare index variable and break out when it reach the end, but that was my first year of python coding.

[@njmanga617]

Great video enjoyed it

[@fiords]

How does it match with the fact that FOR loops are quicker than WHILE? The we come to WHILE implementation with a help of C code?

[@mCoding]

A for loop will be faster than its while equivalent because the while equivalent involves lots of instructions that happen in Python rather than C. With a builtin for loop (in CPython), all the parts that are cookie-cutter have been moved into C, which is going to be a lot more performant. However! All the business about calls to iter/next and raising StopIteration does _really_ happen under the hood (even though it is fast), which can prove useful in situations that are manipulating iterators directly.

[@ZeroIntensity]

iirc for loops dont actually raise StopIteration internally, but instead just iterate until PyIter_Next(iterator) returns a null pointer

[@mCoding]

Good point! There may be optimizations for some iterators that are written in C where they can optimize away the actual creation of a StopIteration if you don't observe it. But for even for builtins if you call next on an exhausted iterator a StopIteration is raised, and if you define your own iterator with a dunder next method, that method should signal that it is done by raising a StopIteration as well. So while technically I think creating a StopIteration can be avoided, I think that semantically it is still supposed to "be there".

[@rusektor]

You could use second parameter of iter: x = next(it, None) Then you could check: if x is None: break

[@twistedsim]

wont work if the iterator contains None, you need to use a sentinel.

[@rusektor]

@@twistedsim "None" is just an example. You could use any other value you don't expect in the collection.

[@twistedsim]

@@rusektor using something like `sentinel = object()` actually guaranty that you never match something else! :)

[@matheusaugustodasilvasanto3171]

Exceptional!

[@mCoding]

I see what you did there!

[@prawnydagrate]

1:57 - The way the loop breaks out messes with for..else blocks, that's all I can think of. EDIT after watching the solution - wow I didn't think of that

[@anonymoussloth6687]

I was just learning about cpp20 coroutines and trying to make a custom for each loop for it and this was helpful I know you do cpp content also so could you do a video about cpp coroutines, concepts, modules, etc introduced in cpp20?

[@mCoding]

I'm actually waiting for better compiler support before making concepts, modules, and coroutine vids for C++! But they are definitely in my head!

[@ABaumstumpf]

coroutines have become suuuuch a convoluted mess in C++20 now... the earlier proposals would have been a lot nicer but now i think we will just avoid them all together. Specially given the absurd misuse of terminology - described in terms of "future" and "promise" but not related to std::future or std::promise!?!? They could have been nice but the WG becomes more and more of an ivory tower creating just purely academical unusable extension.

[@davidardo4466]

Hello teacher I just have those questions Is math important or not? can i score a very good performance.? I like it very much, I hope you will answer me.

[@core36]

In dbpro (not very known at this point in time) a for loop is faster because a while loop does a callback and by default a screen refresh at every iteration. A for loop does only one callback at the end of the whole loop and no screen refresh. In c++, you can replace every while loop with a for loop. You can change the value to check against inside the for loop. But what’s the point? Every loop could be replaced with a while(true) and an if statement. I can use a wrench to drive a nail but sometimes it’s better to just use a hammer. I don’t discriminate if you do this to better understand the inner workings of the language you are using. Just don’t do it in an actual project.

[@mjkaelbling]

for x in expr: print(x) else: print("coming soon in a follow-up video?")

[@mCoding]

Haha I'll take the suggestion!

[@phil6758]

It's actually better to wrap the while loop in a try clause rather than wrapping try clause in while loop with break. E.g. it = iter(range(100)) csum = 0 try while True: rv = next(it) csum += rv except StopIteration: pass

[@Tri-Technology]

Why is that better?

[@sidneydriscoll5579]

You could use index references in a while loop instead of a for loop as well

[@mCoding]

Yes absolutely! In this example we used a list, so indices could be used directly. However, think about what you might have to do if the expr = [1,2,3] was replaced with just some arbitrary EXPR (remember I said ANY for loop could be replaced). In that case, we can't assume EXPR will accept indices. Under the hood Python actually uses this iter/next/StopIteration business, so it's good to know, even though you hopefully will never use it!

[@Shredder2898]

I think the title to this video is a little bit misleading.... Everything is redundant if you view it like that. Better title might have been 'Every python for loop throws an exception'

[@mCoding]

I agree, nearly everything is redundant, but I don't see it as a bad thing as obvious a for loop is more expressive and clear than it's while-equivalent definition.

[@MLGranny]

Meanwhile Golang: While loops are redundant

[@koleso1v]

So, you replaced a 2 line for loop code with an 8 line while loop code. Great job!

[@hjewkes]

I thought this was silly when you said you were going to rewrite for as a while, but TIL about else: on try/except... Well done

[@mCoding]

I fully admit it is still a silly thing to do, though I also think it help understanding of how iterators work under the hood. Glad you learned about else on try/except. Did you know about for/else as well?

[@hjewkes]

@@mCoding I did not! Just read the pythontips article now - there are places in my current project I could use that haha. Do you have a video on it?

[@masheroz]

Do you also know you can put an else clause on a for loop? When I learnt out about it, I found a user case for it about a day later.

[@everettehungerford2858]

So I'm very new to Python and tried my hand at creating loops that printed a list in order and finished without error. I came up with a for loop and a while loop and received no errors or exceptions. What am I missing about this? My For Loop: list = [1,2,3] for each in list: print(list[each-1]) My While Loop: list = [1,2,3] x=0 while x

[@kosmonautofficial296]

interesting video thanks!

[@mCoding]

Glad you liked it!

[@shiro3940]

there are things in this world better left unknown. I'm just a hobbyist and my Python skill is still on the deep underground level. and yes, this is really weird to me and I don't want this "while loop" replacement of "for loop" to attach into my head.

[@overclucker]

expr = [1, 2, 3]; l = len(expr); x = 0; while(x < l): print(expr[x]); x += 1; Have I been doing it wrong this whole time?

[@cyrilanisimov]

World goes mad - I read that in Go “while” loop was replaced with “for” and now you’re telling about redundant “for” loops)

[@oxey_]

This is why I like Rust's `if let` and `while let` statements so much. I can just loop over an iterator and pattern match all values idiomatically until the end of the iterator. I really miss that with python, especially since a lot of other iterator operations like `map` or `reduce` can't just be applied like you would in rust, c++ etc but rather are their own little function. It feels like I'm always better off using a list comprehension compared to using `map`, which feels added like an afterthought

[@Milecarful]

No, it is not an afterthought. It is just encouraged to use comprehension and map, reduce and filter are not removed to preserve backwards compatibility. But they are in a sense mistakes.

[@oxey_]

@@Milecarful oh you're right actually. Still, they could just add them as methods I could chain to my iterator (as a separate function, idk) and just let me supply a lambda, but they don't. It's quite unfortunate imo

[@Milecarful]

@@oxey_ I don't understand why, when comprehension is more expressive and easier to write. Not only that, what you're proposing would violate the Zen of Python. Finally, comprehension will always be faster because of the overhead lambdas have.

[@ABaumstumpf]

@@Milecarful What "zen" of python? That it is an under-specified language that constantly has code-breaking changes? And performance-overhead? In Python? If you care about performance you do not use python.

[@Milecarful]

@@ABaumstumpf The last time Python had code breaking changes was with 3.0. This was more than a decade ago, get over it. The Zen of Python values one, and preferably only one way to do something. In that regard, map, filter and reduce violate it when something like list comprehension exists.

[@stkyriakoulisdr]

There is still a small bug missing i think: What about the for .. else blocks?

[@worgywow]

and then there's golang which completely removes while loops and uses for loops instead where you can make while loop functionality from a for loop

[@robertbrummayer4908]

Interesting video as usual

[@HunterFruitti]

so that's the drawback of having no "hasNext" method for iterators

[@busterdafydd3096]

Should only use exceptions as a last case scenario.. for loops are far more efficient than while loops in most scenarios, and I'm sure it's the same case here

[@no_fb]

a 'for' iterates on an iterator, you should ask yourself the question: how does the iterator tells it has reached the end? doesn't it raise StopIteration too? ;) It's counter-intuitive to people coming from C and other languages, because exceptions are usually demanding and associated with bad performance... it would be interesting to see if that's an impact in Python too

[@gregorymorse8423]

I mean "redundant" is utterly subjective. If you want to make such an argument then "if" is redundant as a while loop could also simulate conditionals. In C everything is redundant, goto and labels are clearly enough. So how to choose base representatives and argue that everything should be expressed in terms of them. For loops were motivated originally from mathematics where the notation of things like sums and products makes a before next iteration statement convenient to represent with the initialization and loop condition.

[@travelthetropics6190]

never knew while loops in Python is so complex, used to write simple while loop syntax with C

[@ericecklund676]

Hmmm...in the last example using a print(it) prints out the addresses in memory of your list objects. Maybe it should be print(x) instead?

[@mCoding]

The last example is intended to show that you can access the iterator within the body of the loop, which was accomplished by printing it. The intended behavior was not to print 1, 2, 3, but to print iterator, iterator, iterator.

[@ericecklund676]

@@mCoding Okay, thanks for that explanation. It just confused me because the previous examples you did were designed to print 1, 2 ,3.

[@jullien191]

오늘도 고마워요

[@subjekt5577]

Question:. Shouldn't it be 'raise StopIterationException()' and not 'raise StopIterationException'?

[@JakeFace0]

Gonna stop feeling so bad whenever I use "try: [CODE] except: pass" to fix indexOutOfRange errors.

[@fgfanta]

I hoped the video was about using the itertools package :)

[@spenhouet]

There is actually a better implementation using the while loop with an iterable: expr = [1, 2, 3] it = iter(expr) while x := next(it, None): print(x) Should still not be used. Just saying, this way it's not as horrible.

[@user-xh9pu2wj6b]

Beautiful, not only is this less readable, it also doesn't properly catch StopIteration. Complements to the chef!