Safe Rust AIN'T SAFE!? (cve-rs)

[@xenotimeyt]

Alright, I've made a Part 2 video that hopefully can address some of the confusion / questions people have. Feel free to check it out: kzbin.info/www/bejne/nJXYZ4KZhr1onac

[@brandonlewis2599]

Not able to play either video. Just get a generic error.

[@xenotimeyt]

@@brandonlewis2599 Huh, it works fine for me.

[@melkii_mel8218]

Exceptionally clear explanation of basic Rust concepts Made it easy to grasp complex foundational ideas in just 20 minutes even with close to no prior understanding 👍

[@xenotimeyt]

Thanks! Really glad it helped :)

[@pmmeurcatpics]

This is the best explanation of co- and contravariance that I've seen!

[@xenotimeyt]

Thanks!!! It certainly was *something* lol.

[@AK-vx4dy]

Probably the best "Rust in 7 minutes" positions on internet, then followed by a grasp of type theory. Respect!

[@Oler-yx7xj]

Using necromancy to extend lifetimes, another thing I didn't think I would see. Great Video

[@AlexKen-zv8mm]

walking _dead

[@snowman4933]

If you said this out loud before a non-technical person, they would've literally started thinking that we programmers are doing dark magic

[@DooMWhite]

Amazing video!

[@xenotimeyt]

Thanks, glad you liked it!!!

[@DooMWhite]

@@xenotimeytGonna check the Bad Apple one now.

[@nonefvnfvnjnjnjevjenjvonej3384]

this is great.. i love rust.. even though these bugs are there, the whole ecosystem is so much more intelligent

[@Skyb0rg]

You touched on it at 18:20, but the Rust Vec type is neither co- or contravariant. This is because you can mutably add elements! If you only read elements it’s contravariant; if you only add elements it’s contravariant. It’s sometimes called invariant, and now I’m curious how Rust handles that.

[@xenotimeyt]

Actually Rust’s `Vec` type is covariant! You can find this here: doc.rust-lang.org/nightly/nomicon/subtyping.html Because of Rust’s borrowing rules you can only have one mutable reference, so it’s actually fine for it to be covariant. If you want it to behave more normally you would have to use an interior mutability type like `Cell`- those actually are invariant.

[@a46475]

​@@xenotimeytThen it should not be. It is not fine. If I define a function that accepts a Vec and you pass it Vec and the function adds a dog to it, what do you have?

[@xenotimeyt]

@@a46475 To add stuff to it you need to have a mutable reference to it (`&mut Vec`). Mutable references are invariant (and this is why).

[@a46475]

@@xenotimeyt oh ok

[@fungod-cm3zv]

great video

[@xenotimeyt]

Thanks, glad you liked it!

[@frankbauerful]

The KZbin algorithm recommended me this video. Probably because I was watching the trial of the armorer who is on trial because of unsafe handling of guns on the set of the movie Rust. Well done KZbin AI 👍

[@aoeuable]

I am now even more convinced that subtyping is the root of all evil.

[@xenotimeyt]

It is interesting given some languages like Haskell don’t have subtyping at all.

[@oglothenerd]

The video sounds like it was sped up by 1.25x.

[@xenotimeyt]

Other ppl said the same thing, back when I made this I sped it up a little because I felt like I was talking too slowly. Haven’t done it after this vid tho.

[@oglothenerd]

@@xenotimeyt I sped up one of my videos, yeah, never again!

[@zookaroo2132]

this turned out to be the simplest way to learn Rust lol

[@xenotimeyt]

:)

[@mohitkumar-jv2bx]

This was great. When i saw this on reddit, i didn't get it. Even the docs were not that useful. This explained the issue very nicelyvv

[@xenotimeyt]

Thanks, glad to hear it!

[@longbranchgooberdapple2238]

Anything explained in terms of cats and necromancers is (obviously) easier to understand.

[@fabiopetrillo]

This video is the proof that Rust is very safe in practice.

[@xenotimeyt]

Hey, I did put "under very specific circumstances usually its fine" in the thumbnail. Obviously everyone noticed that. /s

[@pqnet84]

Interestingly this video has brought enough attention to the original bug report that they had to close the conversation because of spam.

[@xenotimeyt]

It didn't- as far as I can tell the `cve-rs` project got up high on Hacker News. Assuming I'm not misremembering it was locked for spam before my video.

[@_fudgepop01]

2:15 the moment I saw that the first generic numbers typed were 69 and 420 was the moment I knew this was going to be a damn good watch

[@xenotimeyt]

This is the way. ;)

[@AlexMcKenzie-m2m]

To listen to it normally set speed to 75%

[@xenotimeyt]

Yeah I made the mistake of speeding it up a bit since I got nervous I was speaking too slowly. Sorry.

[@a46475]

My understanding of the function "fn weird()" is that 'a and 'b represent arbitrary lifetimes (which can be 'static). The actual values are determined at call time. The caller has to provide arguments that are consistent with the function signature and lifetime constraints. When assign weird to weird_function, the type signature of weird_function should force 'a and 'b to both be instantiated to 'static so that the weird function has the following signature: fn weird_function(_witness: &'static &'static (), borrow: &'static T) -> &'static T If this is the case, then the only problem I see here is that the call "weird_function(FOREVER, borrow)" should clearly be rejected because the argument "borrow" clearly doesn't have a 'static lifetime. Also, the code in the video is rejected by my current compiler ( rustc 1.77.2 (25ef9e3d8 2024-04-09) ). The following is something I found that works: use std::vec; fn main() { let borrow: &'static Vec; { let value: Vec = vec![69, 420]; borrow = bad(&value); } println!("{borrow:?}"); } static UNIT: &'static &'static () = &&(); fn foo(_: &'a &'b (), v: &'b T) -> &'a T { v } fn bad &'static T { let f: fn(_, &T) -> &'static T = foo; f(UNIT, x) }

[@xenotimeyt]

Your intuition about the compiler seeing `weird(STATIC_UNIT, borrow)` and deciding 'a and 'b must both be 'static is correct. That’s precisely why we have to “smuggle” weird. When we say something along the lines of `let f = foo;`, the compiler has to choose the lifetimes 'a and 'b at that point, even though it has no idea about what we actually will call `sneaky` with. So we tell it to set (using your naming) “foo”’s 'b to “bad”’s 'a, and “foo”’s 'a to ‘static. We now have a `fn(&'static &'a (), &'a T) -> &'static T`. This is perfectly correct so far, since calling this new function would require an `&'static &'a ()`, which is only possible to construct if 'a outlives ‘static. This I where we pull some sleight of hand (in your example it’s done implicitly when you use the _ to infer types, I tried to do it explicitly). Since any &’static () is also an &'a (), we can transform this function into an `fn(&’static &'static (), &'a T) -> &'static T`. Which we can hand STATIC_UNIT and unleash chaos. This step is where the logic you describe simply isn’t used. Changing the function from taking an `&'static &'a ()` to taking an `&'static &'static ()` doesn’t involve any arguments. It’s just ordinary contravariance (as described in the video). Hopefully that clears things up, sorry for the confusion. And also with regards to it not running on your compiler, I’ll have to check that out at some point but I am a little busy with other projects (yt and otherwise) right now.

[@a46475]

@@xenotimeyt My problem is that if I can’t call foo(UNIT,x) directly (because x doesn’t live long enough), why am I allowed to call f(UNIT, x)?

[@xenotimeyt]

@@a46475 The reason you can’t call `foo(UNIT, x)` is because the compiler has to pick a 'a and a 'b and in this case they both must be 'static. Since x does not outlive 'static, this is rejected. On the other hand the type signature of `f` is `fn(&'static &'static (), &'a T) -> &'static T`. There are no lifetime parameters for the compiler to pick (the 'a is the 'a passed into `bad`, and we can pass in UNIT and a borrow with no problems at all.

[@pqnet84]

I'm not sure what's more "sus" in this: the ability to create "weird" functions that can leverage type information of arguments to infer lifetime nesting or the covariance of reference to reference lifetimes. Probably the ability to cast an & 'static & 'static to a &'a &'b in a single step is the issue: If you are forced to go through an intermediate step (such as &'a &'static) you'd find yourself banging against the borrow checker fairly soon (i would think you would never be allowed to create a &'static &'b)

[@xenotimeyt]

See my part 2 video - kzbin.info/www/bejne/nJXYZ4KZhr1onac You very much can do it one step at a time, explicitly laying out all the types, and the borrowck is none the wiser. ;)

[@jfb-]

I feel like what's up here is that weird should be a function that has a bound 'a : 'b, which can be thought of as an extra argument that it needs to accept as a proof that a outlives b (just like when a function has a bound that T: Display, you can think of it as accepting the actual implementation of T being Display as an argument). Then the type &'b&'a() isn't actually carrying that proof, it must be carried separately.

[@xenotimeyt]

@@jfb- This is a really good thought- traits are secretly just implicit parameters that the compiler can pass for you. In for example the Scala language it doesn’t have traits natively but you can get them this way, because it does let you mark parameters as implicit and the compiler will figure them out. So the function really needs to accept some sort of proof that `'a: 'b` as an argument. Rust sort of pretends like `&'b &'a ()` does this but it doesn’t. Because any `&'b &'static ()` is clearly an `&'b &'a ()`, but a proof that `'static: 'b` is clearly *not* a proof that `'a: 'b`. I really like that actually. :)

[@Lecopivo]

Imagine a function that takes a proof that `a

[@quickdudley]

@@xenotimeyt Traits could in theory just be implicit parameters that the compiler passes for but that's actually not how rustc implements them unless you use the `dyn` keyword.

[@Otakutaru]

I am not ready for rust. Now every ' in plain text scares me

[@xenotimeyt]

Well this is advanced stuff you probably shouldn’t do lol, I wouldn’t be discouraged to learn the language because some crazy type system issue doesn’t make sense. Btw someone else commented on the part 2 video where I said “'twas” that they thought 'twas was a lifetime lol. ;)

[@noxagonal]

Same... Although I love template metaprogramming in C++ to a point where my code sometimes looks like alien language... So... I guess I'm a little hypocritical.

[@xenotimeyt]

@@noxagonal Hey, I’ve written some heinous stuff with C++ templates, type traits, concepts, and decltype/declval. It’s fun and seems like your making progress but usually I get to the end and realize that I didn’t actually need all that chaos lol.

[@Otakutaru]

@@xenotimeyt I understand. The truth is, that I've never had to deal with lifetimes in my rust journey, is that kind of thing that you know it's there if the situation requires it

[@xenotimeyt]

@@Otakutaru You will one day- and you’ll have the wonderful privilege of not needing to know all the crazy details like you do for this bug. ;) Best of luck on your Rust journey tho! :)

[@comma_thingy]

13:00, your logic on why &'static &'static should be able to coerce to &'b &'a seems shakey. Given the earlier statement that &'b (type with lifetime 'x) means that 'x outlives 'b, and &'a has a lifetime (at most) of 'a, then 'b must be shorter than 'a. So the supposed intermediate step of "&'static &'a ()" should be valid is wrong. Unfortunately, it seems the compiler also seems to miss this, hence allowing 'b to be independent of 'a, even though the function definition should implicitly force 'a:'b

[@xenotimeyt]

If we keep things as they are then, well, yes the conversion is invalid. The thing is that this is simply the result of borrows being covariant in their lifetimes- `S: T` implies `&’q S: &’q T`. So (*if* you don’t implement the proposed fix and you’re forced to have a constraint `'x: 'y`, then we have no problems. The thing is if we run with your version and say that borrows *aren’t* covariant that’s a pretty basic assumption and that would require reworking loads of stuff in the compiler and breaking heaps of existing code. So yes that conversion is incorrect but only if we assume the prior step isn’t. Hopefully that makes sense. :)

[@karpfenboy]

explained it so well that it felt like a really simple conclusion in the end

[@xenotimeyt]

That’s the idea ;)

[@kensmith5694]

BTW: No non-trivial language can be said to be safe. You could write a C interpreter in Rust and include the text of an unsafe C program for it to run.

[@xenotimeyt]

I don’t know about that- there’s no way to make any normal GC’d language segfault- you could write a c interpreter in it sure but you’d never actually be writing to memory you’re not allowed to touch.

[@kensmith5694]

@@xenotimeyt The definition of "allowed to touch" is where you may not have thought about it. The "machine" that is the interpreter would be the machine in the question. Thus it would like you have a system that doesn't even do protected memory.

[@xenotimeyt]

@@kensmith5694 Ok but that definition of “unsafe” isn’t useful- the language can’t possibly know whether you intend for something to count as memory for an interpreted program or not. Nobody would argue that nothing in the real world is safe because you can imagine doing something unsafe and then bam there was unsafety.

[@kensmith5694]

@@xenotimeytThe definition is useful because it explains why on some level no code is ever proven to be safe by the compiler checking it or runtime checking during the testing phase.

[@xenotimeyt]

@@kensmith5694 You can make up whatever definition you want, but a definition that leads you to conclude everything is unsafe is pointless. It gives you no information about a program you apply it to. This is a mathematical fact too: if P(unsafe) = 1 then you have gained -log(1) = 0 bits of information.

[@agungokill]

how to get memory faulty in rust : cannot be concidence. other language : shit i didnt know it it would be faulty

[@xenotimeyt]

python: is it memory safe or not? I don’t know it’s been running for three days and it’s still not done yet

[@kayakMike1000]

Basically... Its not endorsed by JD Power and Associates.

[@therealyojames]

Why does the mic warble in and out of clarity

[@xenotimeyt]

Because the mic was my phone balanced on some random boxes. ;) I’ll get a real mic eventually…..

[@therealyojames]

@@xenotimeyt ahhhh rip, good video otherwise, I couldn’t for the life of me wrap my head around the lifetime tick marker stuff, but thats my fault lol

[@xenotimeyt]

@@therealyojames Well I’m the one whose supposed to be explaining it lol. I tried to go through the basics but did so really quite quickly- the book might be helpful here: doc.rust-lang.org/book/ch10-03-lifetime-syntax.html Glad you still liked it tho. :)

[@therealyojames]

@@xenotimeyt nah you did a good job explaining it, my brain just couldnt absorb it very well because I’m sleep deprived rn 😂

[@xenotimeyt]

@@therealyojames Dw I’m a college student doing a double major- I’m always sleep-deprived. ;)

[@maninalift]

The "witness" is not a sound witness to a outliving b. Since it is satisfied by lifetimes x >= a y >= b x >= y This doesn't proove (ie act as a witness to) a >= b

[@xenotimeyt]

Yes. An `&'static &'static ()` counts as an `&'b &'a ()` even though obviously proves nothing. Really the function definition should implicitly be `fn weird(...) -> ... where 'a: 'b`, but doing that would break things because of the way storing functions in variables (or passing them around) actually works. I explain that side of things in part 2: kzbin.info/www/bejne/nJXYZ4KZhr1onac

[@jongeduard]

I have seen both videos. Fantastic explanation. I have come across earlier people mentioning it, but I did not read the details yet. The thing is, everything has bugs. But the number and significance of bugs strongly differs. In Rust these are both low. You really have to do quite a lot of specific work make this problem show up. So it does not change at all how much I like Rust and it's actually amazing how people do their best to test and experiment with everything. It's part of actually making it all better.

[@qbasic16]

Great video, you earned a sub 😊

[@xenotimeyt]

Thanks! :)

[@eitanseri-levi2169]

I cannot unsee this

[@lepidoptera9337]

No language can save you from your worst self. ;-)

[@emjizone]

Some day we will finally implement mem security at hardware level, instead of fooling around.

[@xenotimeyt]

Implementing this stuff at the hardware level doesn’t guarantee anything will be totally safe either- some ARM chips have pointer authentication but then some people used side channels to crack it: pacmanattack.com I’ve actually played around with microarchitectural security and implemented an attack known as website fingerprinting, where JavaScript on one website uses the latency to do certain operations to guess what’s happening in another tab (usually you use an ML model). It’s not super reliable but the fact that my sketchy version with a simple decision tree and not that much data worked at all is definitely a bit scary. Solving things in hardware is definitely something that could help, especially given all the code in not-Rust, but it’s not a way out at all.

[@mr.togrul--9383]

This is so awesome, you simplified cve rs perfectly, instant sub and like from me

[@xenotimeyt]

Thanks, I’m so glad you liked it! :)

[@zergeyn]

And they say C++ is complicated...

[@theohallenius8882]

Perfect, now I no longer have to fight the compiler, just call extend() everywhere 🤣🤣🤣🤣

[@Matt23488]

Wow. That is really neat. I don't have a ton of experience with Rust, but I completely understood your explanation. Although it is definitely a bug, it thankfully doesn't seem like something someone would inadvertently create in real code, unless they are dealing with complicated lifetime semantics without fully understanding lifetimes...

[@xenotimeyt]

Glad you liked it! You’re right that it seems very unlikely to create by accident, but I hate to say I don’t think you’re wrong about it being introduced by accident being totally impossible either.

[@LuisCabreraT]

If you really want a safe and fast program you should write assembly

[@xenotimeyt]

Ah yes the safest language of them all

[@russianbotfarm3036]

Yeah, it always seemed strange to me that the rust people couldn’t _prove_ it was safe, given that it was designed from scratch, amid sota theory and tools.

[@xenotimeyt]

Yeah, the fact that the language doesn’t even have a formal specification (and they only recently started on one) has always been concerning to me in much the same vein. At least there is an effort on that front ( github.com/rust-lang/rust/issues/113527 ), but it isn’t great that we don’t even have a draft at this point. I know “probably safe” is very much not easy but the way Rust is supposed to be this rock-solid safe thing it definitely seems off to have so little in the way of formality. Just my thoughts tho.

[@vladomaimun]

Ok but why doesn't rustc complain about the argument borrow not living as along as 'static when weird_function() is called? Does it check the types against the definition of weird() or against the definition of weird_function()?

[@xenotimeyt]

We aren’t substituting 'a for 'static, we’re just using subtyping. It’s fine to convert an `fn(Animal, Animal)` to an `fn(Cat, Animal)`. If you want to see it step by step also there’s a part 2 video: m.kzbin.info/www/bejne/nJXYZ4KZhr1onac

[@nordgaren2358]

It's such a pain in the ass to write a buffer overflow in this language, the cve-rs repo needed an issue for it's cve (It was not always generating a buffer overflow on every target)

[@tee1532]

At 0:49 you say "this is an incredibly hard problem to solve". I wish your video explained why! It looks like it's easy to fix the example shown in the video and GH issue. The line: let weird_function: fn(&'static &'static (), &'a T) -> &'b T = weird; seems to obviously require that &'a is at least as long-lived as &'static, because otherwise it would not be compatible with weird's signature. According to weird's signature: fn weird(_witness: &'b &'a (), borrow: &'a T) -> &'b T { It is obvious that the lifetime of weird's second argument must be the same as or longer than the lifetime of weird's first argument. So in weird_function's type let weird_function: fn(&'static &'static (), &'a T) -> &'b T = weird; the second argument &'a T must have a lifetime longer-or-the-same as the first argument which is &'static. So why does the compiler allow calling weird_function with a second argument that has a lifetime shorter than &'static ?

[@xenotimeyt]

I’m not sure I understand how the `let weird_function: … = weird;` line requires that 'a is at least as long as 'static. All that needs to happen is any `&'static &'static ()` needs to be an `&'b &'a ()` which of course is the case. This is function arguments being contravariant enables: We’re allowed to convert a `fn() -> …` to a `fn() -> …`. In this case `&'b &'a ()` is the general type and `&'static &’static ()` is the specific type. We aren’t trying to set 'a and 'b to 'static or anything, we’re creating a new function that does the same thing with a different type signature. If it helps imagine that I instead do this: ``` fn weird_function(witness: static &'static (), borrow: &'a T) -> &'b T { weird(make_specific(witness), borrow) } fn make_general(witness: &’static &'static ()) -> &'b &'a () { witness } ``` Hopefully that example should clearly be all above board. Contravariance is sort of a shortcut to do the same thing without actually creating a new function. That’s why contravariance is a key part of the trick- the whole `&'b &'a …` thing is normally checked at the call site (in the code above when we call `make_general`. But here we never actually have to call it. In the code above you can see that `weird` and `weird_function` *do* the same thing. So the language is perfectly happy allowing us to convert `weird` to `weird_function`. You don’t need to have a `fn make_general(cs: CatSoul) -> AnimalSoul …` and then call it explicitly. You can just convert a `fn(AnimalSoul) -> …` to a `fn(CatSoul) -> …`. Hopefully that clears it up a bit. As for explaining why fixing it is difficult that’s sort of a difficult thing to do since everyone has their own slightly different idea on how it could be fixed easily. The reason it’s hard is because there isn’t a way to fix it easily, and I video going through all the possible easy fixes that wouldn’t actually work wouldn’t be very interesting. I *might* make a follow-up video explaining one or two not-quite-correct easy fixes and then the sort of ideal way to fix it. But I’m pretty busy both with the next vid and lots of other stuff. There have been a couple of people in the comments seeming to specifically think that you’re somehow substituting 'a and 'b when you don’t. Let me know if I’m misunderstanding you though.

[@tee1532]

​@@xenotimeyt I thought the original function signature fn weird(_witness: &'b &'a (), borrow: &'a T) -> &'b T { specifies &'a is at least as long as &'b. I thought you wouldn't be allowed to use contravariance to specify incompatible bounds. E.g. if the signature were fn weird(_witness: B, borrow: A) -> B { I'm pretty sure fn extend(borrow: A) -> B { let weird_function: fn(SuperA, A) -> B = weird; would not be allowed by the compiler. I just don't see the difference between this and writing the same thing with lifetimes instead of types.

[@xenotimeyt]

@@tee1532 The original signature should specify that exact constraint but currently doesn’t. This is the real problem- the language doesn’t actually have away to associate these kinds of constraints with a function type. Ideally you’d be able to say the type is `for where 'a: 'b fn(&'b &'a (), &’a T) -> &'b T`. But currently putting a where clause there isn’t a language feature that actually exists. That’s the bug and the ideal way to fix it. Hopefully that makes sense. :)

[@tee1532]

@@xenotimeyt why is it an ideal fix to require the programmer to type extra words? The compiler should currently be able to infer that constraint anyway

[@xenotimeyt]

@tee1532 (See comment after this one first, I think this one isn’t actually answering your question) How can it infer the constraint? Are you saying that at the time we use `weird` to a `fn(…) -> …` trait object the compiler should look through the type of `weird` for any of these constraints? That seems like it would work but the problem is we don’t know what 'a and 'b are at that point. The compiler deduces the type of `weird` to be `for fn(&'b1 &'a1 (), &'a1 T) -> &'b1 T`. This is basically just abstracted over any possible lifetimes 'a1 and 'b1 ( doc.rust-lang.org/nomicon/hrtb.html ). So we can’t check it then since we don’t know what 'a1 and 'b1 are and after that the constraint 'a: 'b is nowhere to be found. Unless we somehow store it in the type. Sorry for the confusion. Honestly I’m sort of regretting not explaining the `for` bit in the video but I just realize how important it kind of just seemed like unnecessary having to explain another weird rust feature. But I think it’s clear that’s not the case.

[@meqativ]

18:16 the cops are already coming after you

[@sudonick-kn5zn]

dude you explained the cve and lifetimes so well! learned a lot from this video.

[@xenotimeyt]

Glad you liked it! :)

[@nexpro6985]

It sounds like you speeded up the video.

[@xenotimeyt]

Just a bit- I felt like I was talking too slow but I’m not going to keep doing that or anything.

[@Tombsar]

@@xenotimeytPlease don’t speed it up like this in future videos. I found it very off-putting and hard to understand in places.

[@xenotimeyt]

@@Tombsar I won’t.

[@henrycgs]

when I saw the code compile I was just.... fuck. oh, fuck. how on earth do you even fix this? is it even possible?

[@michawhite7613]

The compiler will need to encode outlived constraints into the types of variables

[@henrycgs]

@@michawhite7613 yeah, after a bit of thought I imagined something like that. I wonder if it's possible without additional syntax.

[@xenotimeyt]

Oh boy do I have the part 2 video for you: m.kzbin.info/www/bejne/nJXYZ4KZhr1onac

[@feuermurmel]

Can you add the part to the URLs in the video description? Otherwise KZbin won't recognize it as URLs (at least in the iOS app). Thanks! 😊

[@xenotimeyt]

Done! :)

[@ggsap]

If you are speeding up your voiceover could you uncheck the pitch option? Quite annoying

[@razvanfilea8076]

Is this going to be fixed by the polonius project?

[@xenotimeyt]

Polonius (NLLs) is about making the lifetime system *more* accepting. If you’re curious to learn more and also understand how to fix it boy do I have the part 2 video for you: m.kzbin.info/www/bejne/nJXYZ4KZhr1onac ;)

[@parlor3115]

I'm not worried about these issues, because in order to invoke these exploits, you'd actually have to understand Rust which only like 2 people do (out of the entire user base that is 5 people). The only other option is installing libraries authored by those 2 people, but you can't do that as a malicious actor since Rust is a server-side language.

[@xenotimeyt]

*sigh* Sure Rust isn’t that popular but it’s been growing fast and it’s definitely more than just a few people. And yeah it’s not a serious security exploit- but it’s still a problem. And I’m not sure what server-side has to do with it not being a problem, if this ends up in some library that your server users now you’re back to the same world as broken c code where you can read loads of uninitialized memory and probably rce if you’re clever about it.

[@aronhegedus]

really good explanation, I've done a bit of rust and felt like I understood this! thanks for the video

[@xenotimeyt]

Thanks for that, glad you liked it!

[@sansmojo]

Alright, I guess I'm slow. I followed up until you started talking about covariance and contravariance. It seems that you're saying that any function that can take an animal soul is a function that can take a cat soul, and so any function that can take a cat soul is a function that can take an animal soul. That makes a bit of sense to me, in that a cat is an animal, so a function that can take a cat soul is a function that can take an animal soul, but that's not really true because it can't take any animal soul; only a cat soul. What am I missing? Maybe another way to talk about it is a vending machine that can accept any US currency (coins or bills) and a vending machine that can accept coins. The vending machine that can take any US currency is a vending machine that can take coins, but the vending machine that can take coins is not a vending machine that can take any US currency (it can't accept bills).

[@xenotimeyt]

It’s only one way; fn(AnimalSoul) is an fn(CatSoul) but not the other way round.

[@whythosenames]

They just need to check which function instance is really used when doing such a cast in extend, i.e. the exact values for a and b. Here 'a has to be more general than the 'a provided to extend and to 'static, so for example exactly equal to the provided a. For b, 'b has to be more general than 'static, but from the output, more specific than the provided 'b. Then they could easily check if the function signature is valid, also for other conditions that have to be met (maybe there are others but I am not a rust developer). But that only works if b has a shorter lifetime than a, which is what we want

[@xenotimeyt]

They can’t, see part 2: m.kzbin.info/www/bejne/nJXYZ4KZhr1onac

[@xshady2967]

finally someone (maybe not first one) is shedding some light on rustc soundness bugs here on youtube, thanks

[@RootsterAnon]

Wow, the amount of info in this video is wild. Thank you for documenting and sharing this with us, it was a nice read to know about.

[@xenotimeyt]

Thanks, glad you enjoyed it! :)

[@jacobesplin20]

Great Video!

[@xenotimeyt]

Glad you enjoyed it! :)

[@helenmachelen4200]

Preemptively absolving rust language developers is weak

[@xenotimeyt]

It wasn’t preemptive….

[@jpjude68]

sounds like a possible solution to the lifetime problem could be applying the idea of a hard link instead of what seems like a symbolic link : any number of instances can reference a bloc of data, but the data is dropped if there are no references remaining

[@xenotimeyt]

It sounds like you’re describing reference counting to manage memory. This is a thing (it’s what swift does), every pointer also has a reference count, which is incremented when a reference is created, decremented when the object goes out of scope, and when it’s zero you free the object. You can create a reference counted pointer in Rust with the `Rc` type. Reference counting everything (like swift afaik) has pretty crappy performance, Rust is trying to get the same speed as C and C++. All this incrementing/decrementing takes time, and more importantly it also is very cache-unfriendly (you can google more I don’t think I can explain it in a yt comment if you don’t know). But regardless, the point is that reference counting is slow- it can actually be slower than full on garbage collection like most languages have. Hopefully that clears things up a bit. :)

[@wasdasdasdcasdc]

Which vscode color scheme is that?

[@xenotimeyt]

Moegi Dark :)

[@haraseesgill8491]

I’ve been sharing this with others. You’ve done a great job covering this, and thanks for sharing such a fascinating problem in a very digestible way. Hope your channel keeps growing… I’d love to keep learning and your style is really well done and organized

[@xenotimeyt]

Thank you so much! :)

[@richardmorgan7734]

I am wondering, what are potential ways this can be fixed? Everything seems legit, and I do not think adding to the compiler "check for these series of functions" would be a sound solution.

[@richardmorgan7734]

nevermind, i see your second video lol

[@xenotimeyt]

I made a part 2 video which explains the ideal fix ( kzbin.info/www/bejne/nJXYZ4KZhr1onac ), BUT basically you want to be able to encode the constraint that 'a: 'b in the function type itself.

[@kfftfuftur]

Ultimately I think the bug is, that you can only replace &'b &'a () with &'static &'static () if it is a valid type to begin with. If 'a doesn't outlive 'b then &'b &'a () will be equivalent to infallible since no valid reference can outlive the value being referenced and therefore it has no valid representations on runtime. To fix this the compiler would in addition to checking if &'static &'static () can be treated as &'b &'a () also need to check if any lifetime conditions that arise along the way are met.

[@xenotimeyt]

Yes that’s right but the trick is how do you remember those conditions. Really the type of `weird` should be something like `for where 'a: 'b fn(…) -> …`. But that type currently doesn’t exist, so you’d have to add that to the language.

[@jonnyso1]

At least I think people are generally aware of that considering the 2023 Rust Survey results.

[@steveoc64]

Thats barely half a page of code, with a really detailed explanation of the bug, and its still really hard to see that it can be doing the wrong thing. Imagine some dystopian future where Rust is in mainstream use in corporate applications, and this stuff comes up for code review ? You can bet that anyone reviewing this code will just say "It passes the borrow checker - LGTM" and clicks the green button

[@sunofabeach9424]

sounds just like any other program written in C++

[@ultrapoci]

Have you ever read a big C code base? It's so full of macros and weird syntax that it is basically impossible to parse. You already live in the world you describe, mate.

[@xenotimeyt]

I mean yes that's right. I like to for example use the token-pasting operator (##) in C/C++ to make macros for timing, where I can say `STARTTIMER(Something)` and `ENDTIMER(Something)` and the macro will create the `timerStart_Something` and `timerEnd_Something` variables. It's truly horrendous but especially when I'm just curious and want to try something I do it anyway. We very much do live in that world but (at least imo) part of the point of Rust is to get us out of there. Also I feel like Rust definitely gives you a sense of security where you don't even think about memory issues as a possibility (assuming you don't use unsafe), and if that becomes a *false* sense of security I would say that's definitely a problem. That's all.

[@ultrapoci]

@@xenotimeyt Well, the presence of this bug is a problem, and hopefully gets resolved sooner rather than later. I don't think false security is a problem, tho. This can be said of any language, since any compiler can have bugs. Programming in C doesn't make a compiler bug less problematic just because C makes less guarantees. Of course, C and C++ have had years and years of people working with them and resolving bug. Also, this is the only known bug in the Rust compiler regarding safety, as far as I know. You still get a ton of benefit using Rust, and it's proven by the people actually building software with it.

[@sunofabeach9424]

@@ultrapoci the problem is that this bug exists from 2015

[@whtiequillBj]

is it known when this bug was introduced? or to rephrase the question is it known if this bug was in a revision of Rust prior to 2015? If it wasn't can the good code not be pushed into the current version?

[@xenotimeyt]

If you want to learn about how you might fix it (it’s not quite so easy)- there might just be a part 2 video on the subject ;) kzbin.info/www/bejne/nJXYZ4KZhr1onac

[@samuelwaller4924]

This is an amazing video. Love the NecromancerWhoLovesCats

[@xenotimeyt]

Thanks! Glad you liked my questionable explanation lol. :)

[@JorgetePanete]

I don't remember if I wrote this in another video of yours The video/voice is sped up

[@xenotimeyt]

Yes. Not something I’m going to keep doing.

[@kacper7516]

i like how u explain things.

[@xenotimeyt]

Glad you liked it :)

[@doce3609]

This is an amazing explanation

[@xenotimeyt]

Thanks, glad it worked for you! :)

[@nevokrien95]

I love the video you really build up to it well. Who's bright idea was it that's every lifetime is a subtype of static? Like if you think about it for a bit that's not true as u have clearly showen. If they wanted a generic lifetime they could have used any.

[@xenotimeyt]

Thanks! It’s not that every lifetime is a subtype (sublifetime really) of 'static - it’s that 'static is a subtype of every lifetime. But yeah, this is essentially a hole where that’s not the case - nested references hide additional constraints you’re dropping. Again tho ideally (and if you don’t want to break existing code) those extra constraints should be separate from the reference type itself. Then it would be totally correct to say &'static T is a subtype of &'a T for any T and 'a.

[@guelakais1438]

It's hard to reproach the maintainers of Rust here, which you don't do either. Rust has some unorthodox features and of course they don't always work. Either Rust or one of the following programming languages will show a proper implementation of these features. At the end of the day, programming languages and the features implemented in them are part of a continuous development process. What today's programming languages cannot do, the following ones will.

[@xenotimeyt]

I do hope that this gets fixed and I fully believe that eventually they’ll get around to it. It’s just that a ton of people see this as something with an “easy fix” and I wanted to clarify that that just isn’t the case here.

[@binary_gaming113]

Great explanation!

[@xenotimeyt]

Thanks, glad you enjoyed it!

[@ukyoize]

Rust sure is weird

[@scottramsay3671]

Thanks for the clear explaination. After reading about cve-rs on hackernews, it seemed like the importance of it all really rested on if it would be possible or not to organically stumble upon bugs like this. (Or if you'd only encounter them by fuzzing or trying to overwhelm the complexity of stuff). This reminds me of when I was learning Haskell and I wrote some code 'f [x] = ...' thinking it would mark x as an array and not realising that it was a pattern match for an array with a single element x in it. Once I figured out why it wasn't compiling, I felt a bit silly. Clearly I was speaking natural language, not Haskell. From that experiance I can 100% imagine myself organically stumbling across this bug while learning rust for the first time. Especially as a former functional programmer who won't bat an eyelid at functions being assigned to local variables, and making their types more generic.

[@iliya-malecki]

i feel like fn(&'b&'a(), &'a T) -> &'b T shouldnt be coercible to fn(&'static&'static (), &'a T) -> &'b T since, while &'static&'static is a valid &'b&'a, it is not a valid &'b&'a in a function that couples the lifetime with some other parameter "borrow: &'a". It implies that "borrow" is also &'static and doesnt check for that, even allowing the second parameter to be &'a T

[@iliya-malecki]

So the real question is why are functions are coercible if every parameter by itself is coercible, if, clearly, a dependency between parameters should be respected?

[@xenotimeyt]

I made a part 2 video which (I think) explains what you’re struggling with: m.kzbin.info/www/bejne/nJXYZ4KZhr1onac Hope that helps. :)

[@milasudril]

I guess these lifetime annotations are more like attributes. In C++, a function pointer must otherwise be an exact match, both with respect to arguments and return type. However, covariant return types are allowed in certain cases when using virtual.

[@hrmny_]

You should invest in a better microphone if you're planning on making more videos Audio quality is often more important than video quality

[@xenotimeyt]

I’m aware- I was just using my phone as it’s the best I have. I will get a real mic at some point, but I mean that’s a good chunk of change.

[@rupen42]

Great explanation! I fully expected not to understand anything but I could follow everything!

[@andreaszetterstrom7418]

Well, at least you have to really work to break Rust. In C or C++ it'll break due to lack of effort instead 😛

[@ernststravoblofeld]

It turns out, there's no language that can keep people from writing shit code. I'm shocked!

[@somenerd8139]

why do you speed up your vids? I’d rather just listen to it at normal speed than not understand anything with a higher speed.

[@xenotimeyt]

I just thought that I was talking a bit too slowly. Not something I’m going to do moving forward.

[@呀咧呀咧]

which theme and font are using?

[@xenotimeyt]

I’m using the Moegi Dark theme, as for the font it’s just the default which I think is Droid Sans Mono.

[@Stopinvadingmyhardware]

Not verified, not good enough to be used in Industrial or government purposes. ADA is superior.

[@xenotimeyt]

Well afaik it’s possible to have memory safety bugs in Ada code too, I think? (Not super familiar with Ada) I definitely have my fair share of reservations about the state of rust in the sense of “not verified”, there are projects like RustBelt but these are very much works in progress and come on guys there isn’t even a specification (yet). That said “industrial or government purposes” is a ludicrously large umbrella. There’s plenty of code for “industrial or government purposes” written in C/C++. Memory unsafety - especially in the case where you really have to *try* to get memory unsafety - I don’t think should immediately disqualify a language from something that broad. I think Rust right now definitely could be used for at least some “industrial or government purposes”. Doesn’t mean I think it’s always the best choice or there isn’t loads of room to improve (seriously guys we need a specification), but I just think that’s way too broad a category to discount Rust from. Of course that’s just my thoughts/opinions, hopefully you can see where I’m coming from here. :)

[@Azyo64]

Very interesting issue. One that should really deserve a fix. But the problem clearly come from the line: let weird_function: fn(&'static &'static (), &'a T) -> &'b T = weird; // => 'a and 'b should be deduced to be 'static Maybe the bug exist because of lifetime, but it should be possible to do the same thing with types, so lets take your necromancers instead: fn chimeraNecromancer(a1: Soul, a2: Soul) -> Chimera; let weirdNecro : fn(Soul, Soul) -> Chimera = chimeraNecromancer; // => Animal should only be Cat! let catFish = weirdNecro(nemoSoul, myCatSoul); // error nemoSoul is not a Soul! The exact same problem exist with types! I tested and rust give me an invalid cast on chimeraNecromancer. So the same logic should be used for lifetime as for types, and that problem would be solved. There might be other variants that are more complex, but that one shouldn't have waited so long before being addressed (when it is). (Instead, people have been putting a lot more effort into using this this bug to break rust)

[@xenotimeyt]

Lifetimes don't behave like types in a very specific way. I made a Part 2 video that explains this in more detail (including the specific way in which types and lifetimes behave differently). kzbin.info/www/bejne/nJXYZ4KZhr1onac

[@DeathSugar]

I wonder if it works on pre-nll Rust

[@xenotimeyt]

The issue was opened in 2015 and NLLs were stable in 2022 so I’m pretty sure it did. :)

[@DeathSugar]

@@xenotimeyt cranelift, gcc?

[@xenotimeyt]

I’m not familiar with cranelift and am not sure what you mean by gcc? I don’t think it depends on the compiler backend since it’s a weakness in the type/lifetime system as it currently exists.

[@DeathSugar]

@@xenotimeyt cranelift uses sophisticated data types to represent everything in rust typesystem, so some states theoretically cannot be represented and compiled by cranelift. gats let them represent read only/write only pointer types which cannot be casted in each other, for example. idea was featured in some twir issue, if I'm not mistaken it to some other project. gcc also has their own rust compiler but I'm unsure how featureful are they, so I wonder if it would even compile with it.

[@xenotimeyt]

@@DeathSugar Ah, I guess in that case it could be possible for them (cranelift) to add these kinds of constraints then. Thanks for explaining it! :) As for gcc your guess is as good as mine.

[@NStripleseven]

That’s… uhhh… that’s wild.

[@mikkolukas]

Your sound is bad - like it has gone through too much compression

[@xenotimeyt]

I still don’t have an actual mic lol, I’ll get one eventually….

[@jtw-r]

it doesn’t really sound like compression… it just sounds like it was slightly spread up in a lossy format

[@coder0xff]

So Is the fix that when changing &'b &'a to &'static &'static in one part of the function type that all occurrences of 'b and 'a have to become 'static as well? fn(&'static &'static, &'static T) -> &'static T

[@xenotimeyt]

Not quite- we aren’t substituting 'a and 'b but we’re just using the subtyping relationship. It’s like converting an `fn(Animal, Animal)` to an `fn(Cat, Animal)`. We aren’t substituting `Cat` for `Animal` we’re just narrowing a type that’s in a contravariant position. I made a part 2 with more detail (which kind of is necessary to understand how to fix it): kzbin.info/www/bejne/nJXYZ4KZhr1onac Hopefully that helps clear things up.

[@Speykious]

Good video, I like the analogies :D

[@xenotimeyt]

Thanks!

[@xenotimeyt]

Hold up I only just realized that you're the one who actually made cve-rs lmao. I swear I was just like "that name seems familiar" and then moved on. Thanks for the kind words *and* for making the crate! :)

[@Speykious]

​@@xenotimeytYou're welcome haha. To be completely honest I only did a third of the work (for example Creative0708 iterated on all the safe transmute implementations) and the license we chose mostly reflects the amount of understanding we actually had when making this. To me there was a problem mostly with lifetimes, but you explained how variance plays into it and I can say you made me understand the exploit better than I did before lol

[@xenotimeyt]

@@Speykious Thanks for clarifying, but also it’s really quite cool to hear that I made it make more sense to even you lol. :)

[@downthecrop]

Good vid brother 🙀

[@icemojo]

We need more videos on the Internet like this. At the end of the day, Rust is ultimately a tool. A tool which brings many great ideas and benefits, even though it IS NOT a "one size fits all" tool. People should realize this more often. The problem with the Rust community these days (especially on KZbin) is, wherever corner you turn towards, Rust is being treated like it's some sort of second coming. It's just a tool, people, it's just a tool. Use it where appropriate.

[@xenotimeyt]

Very much agree. :)

[@diadetediotedio6918]

This is a very specific way to do things But also, it feels like an intuitive error, why would 'a be transmuted to 'static just because you called weird_function passing FOREVER as the argument? The compiler should have rejected passing 'borrow' as the second argument because 'a of borrow is externally defined by the caller and FOREVER is externally defined by a static variable, so any 'a infered by FOREVER locally on weird_function while calling it with FOREVER as the first argument should be objectively different from the 'a from 'borrow' being passed in the same function. This by itself should not be already a reason to reject the call entirely?

[@xenotimeyt]

There's only one 'a which is the one passed to the `extend` function. For simplicity let's call the lifetimes 'a and 'b in the definition of `weird` 'a1 and 'b1. So pretend we said `fn weird(...) ...`. When creating `weird_function` the compiler needs to see if we're allowed to convert a `for fn(&'b1 &'a1 (), &'a1 T) -> &'b1 T` to a `fn(&'static &'static (), &'a T) -> &'b T`. The weird `for` bit is called a Higher Rank Trait Bound or HRTB, and basically just says "for any 'a1 or 'b1 you give me ...". The compiler chooses 'a1 to be 'a and 'b1 to be 'b, and so now we just have to convert `fn(&'b &'a (), &'a T) -> &'b T` to `fn(&'static &'static (), &'a T) -> &'b T` which it can. So now we call the function, the compiler checks that `FOREVER` is a `&'static &'static ()` which it is and so that part is fine. And then it checks that `borrow` is a `&'a T` which it so we're all good. So the choice of 'a1 and 'b1 (which in the original code are 'a and 'b inside the definition of `weird`) is done explicitly by us when we say `let weird_function: ... = ...;`. The 'a and 'b in `fn(&'static &'static (), &'a T) -> &'b T` refer specifically to the 'a and 'b for the `extend` function. Let me know if that doesn't make sense (this stuff definitely is weird). :)

[@tee1532]

​@@xenotimeyt It doesn't make sense to me. The line: let weird_function: fn(&'static &'static (), &'a T) -> &'b T = weird; seems to obviously require that &'a is at least as long-lived as &'static, because otherwise it would not be compatible with weird's signature. According to weird's signature: fn weird(_witness: &'b &'a (), borrow: &'a T) -> &'b T { It is obvious that the lifetime of weird's second argument must be the same as or longer than the lifetime of weird's first argument. So in weird_function's type let weird_function: fn(&'static &'static (), &'a T) -> &'b T = weird; the second argument &'a T must have a lifetime longer-or-the-same as the first argument which is &'static. So why does the compiler allow calling weird_function with a second argument that has a lifetime shorter than &'static ?

[@korigamik]

man this is a cool video and the style. Can you tell me what you use to create the video, the editor stuff, voiceover and tools that you've used?

[@xenotimeyt]

Thanks! There really wasn’t much too it- I just screen-recorded vscode in zen mode for the actual coding parts, and then I used kdenlive to edit everything. The VO was recorded on my phone resting on top of a bunch of small boxes so that it was just above like where my laptop webcam is.

[@harriehausenman8623]

the live editing with all the suggestions and highlighting is super confusing. cant watch it. and the negative biased ui for sure doesn't help.

[@xenotimeyt]

I mean in retrospect I probably should have turned off suggestions, but would I have liked to have everything be animated and only have what’s needed on screen? Yeah. Do I right now have the skills and time for that? Unfortunately not really. I hear you but at least for a noob like me that would have taken ages….

[@NerosCreations]

Great video. Simple fix: when defining the weird_function variable, it seems that the compiler should be able to infer that 'a and 'b must both be 'static based on the lifetime of the witness variable. It should then only allow calls to extend when the argument has lifetime of 'static. -- Essentially the compiler should change the type of extend under the hood, so that it becomes fn(&'static T) -> 'static T. Why wouldn't this work?

[@xenotimeyt]

Glad you liked it. :) The compiler first has to figure out the type of just `weird`, which it can happily infer as `fn(&'b &'a (), &'a T) -> &'b T`. Then when we define `weird_function` we aren’t saying “okay change 'a and 'b to 'static”. We’re saying “hey can I give you an `&'static &'static ()` instead of an `&'b &'a ()`” and this is totally fine. We aren’t ever defining 'a or 'b inside the function, when you say `fn extend…` you’re saying this function will work for any 'a and any 'b (chosen when you call it). But the compiler knows that a `&’static &’static ()` will always count as a `&'b &'a ()` no matter what 'a and 'b are. Hopefully that made sense, let me know if it didn’t or you have other questions. :)

[@AlexKen-zv8mm]

@@xenotimeyt If possible start Rust tutorial series like crust in rust by Jon Gjengset. You have lots in your head that other would like to hear about it.

[@xenotimeyt]

@@AlexKen-zv8mm Thanks for the suggestion, I just might give that a try.

[@AlexKen-zv8mm]

@@xenotimeytyou should , take your time, enjoy it.

[@bujitself]

You missed the transmute part, but cool vid nevertheless

[@xenotimeyt]

I was actually going to add that but the video was getting kind of long and I wanted to get back to working on the next one, that’s all. Maybe I’ll make a little bonus video with it if I have some time- thanks for the feedback tho.

[@hunter-tm2kl]

@@xenotimeyt this was a really great vid, would love to see another on transmute when you have time!

[@xenotimeyt]

Ask and you shall receive (sometimes): kzbin.info/www/bejne/p6LKdah7npyCi5I

[@xenotimeyt]

Ask and you shall receive (sometimes): kzbin.info/www/bejne/p6LKdah7npyCi5I

[@younesskafia4189]

That was a really fun video to watch, very informative!

[@xenotimeyt]

Thanks! :)

[@torsten_dev]

So why is the fix not just that if you use contravariance to extend a lifetime argument that appears somewhere covariant like the return that that lifetime also has to be changed?

[@xenotimeyt]

A couple things: - Returns are covariant not invariant but I’ll assume that was a typo - The actual issue here isn’t that you can do contravariance with lifetimes (although disallowing this was proposed as a short-term fix) - I’m not sure what you mean by “the lifetime has to be changed”? Changed how? The lifetime is “changed” in the sense that it’s converted from `&'b &'a ()` to `&'static &'static ()` which is totally fine, but the hidden constraint that `'a: 'b` was dropped. How would you change the lifetime to keep that constraint alive? Sorry if things were confusing.

[@torsten_dev]

@@xenotimeyt I'm beginning to see why this hasn't been resolved in a decade. Is there a proposed alternative fix?

[@xenotimeyt]

@@torsten_dev If you read through the issue you can see that the sort of ideal solution is to allow constraints to be added to function traits and then to have these constraints be automatically generated as necessary. The type of `weird` is currently `for fn(&'b &'a …) -> …` but the idea is to change it to `for where 'a: 'b fn(&'b &'a …) -> …`. The `for` bit exists and is called a Higher-Rank Trait Bound or HRTB. Basically it just says “hey when you actually use this type you gotta give me a 'a and a 'b.” The proposal is to add `where` clauses to this expression. Then when you actually sub in 'a and 'b from the `extend` function you would know to check `'a: 'b`. Of course this already involves creating a new language feature that extends an already-complicated language feature (HRTBs). You would even (hypothetically if it were implemented) be able to have lifetime where clauses inside of type where clauses like: `… where T: for

[@Turalcar]

@@xenotimeyt I remember wanting to use HRTBs for an associated type trait A where 'b: 'a }. (type B is only defined for lifetimes that outlive 'a) and it didn't work but I didn't expect the current implementation to be unsound. I would probably prefer if they wouldn't allow dropping implied constraints and went with the largest supported type instead.

[@taishi-sama-tfc]

​@@torsten_dev developers said that solution of this problem relies on features of new rustc type checker which is still in development.

[@gordonfreimann]

i feel sorry for rust devs

[@TCMx3]

really enjoyable discussion of the issue. I will say I didnt expect to see exactly that cat reference, but I dont hate it lol