"Bush hid the facts" Bug EXPLAINED

[@nielot0330]

I love computers.

[@matt.w]

Same

[@dominiksalata4234]

same

[@polandman07]

Same

[@FlyTechVideos]

Same

[@lelanceflamer]

same

[@catomajorcensor]

The question we should be asking is "how did Windows developers come up with the worst way to detect Unicode?"

[@ОннокорОктябрь]

- So... how we are going to detect unicode? - -Meth- Math

[@RDCST]

@@ОннокорОктябрь Meth?

[@Cyberfishofant]

​@@RDCSTi was gonna _crack_ a joke, but that wouldn't be funny

[@FriedMonkey362]

I mean it was good enough because it "rarely" occured, still the amount of false positives is indeed too much and yeah it was pretty bad

[@intron9]

It acts like a detector for windows's "Unicode" files that only have English characters, but with a lot of false positives.

[@avitus27]

I can't believe it took me this long to realise: this channel is a fly explaining bugs.

[@Axcyantol]

wait what

[@amberwingthefairycat]

@@Axcyantol A fly (the type of insect) explaining bugs (computer errors, but the word also means insect)

[@TDXC------]

He also... Well creates bugs too (By that I mean destroying windows)

[@Axcyantol]

@@amberwingthefairycat no i understood it, i was just surprised about it being a fly explaining bugs

[@amberwingthefairycat]

@@Axcyantol Oops, sorry, haha.

[@shockwave952]

15 years ago I made a "Windows XP Easter Eggs" video featuring this bug, and now I feel this strange sense of satisfaction finally knowing why it happens. Thanks FlyTech!

[@Dog_Dogs]

That video is best video.

[@elesqueleto2010]

i think you made the video a little too popular, still, good one 👍

[@TwoNumbahNiens]

I can't believe George W. Bush would do this.

[@alexandermorozov8593]

"W." stands for WordPad

[@JeyPeyy]

Amazing how he started hiding the facts at least 7 years before it happened

[@blueschnabeltier_]

Now its George L. Bush, 'cause he hid the facts

[@johndododoe1411]

George Walker Bush Sr. ruled 1089 to 1992, and did secret government stuff with Nixon and Reagan .

[@dylanbksp]

do you know what i can believe bush did?

[@MadsterV]

Note: What Windows Write calls "unicode" is UCS-2, a now-obsolete 16-bit encoding. What today we call unicode is usually UTF-8, a variable-width encoding that conveniently matches US-ASCII, though there's also UTF-16 and UTF-32.

[@billy65bob]

There's not just those variants either. There's the obsolete UTF-7 too (for some reason), and the multi-byte encodings come in both Little Endian and Big Endian flavours.

[@freyja5800]

@@weakspirit_ internally, yes. but when saving text it does default to utf8, since a text in 7-bit ascii and in utf8 are the same, and since in english you rarely need characters beyond that, using utf8 is more space efficient

[@laurentverweijen9195]

UCS-2 and UTF-16 is more or less the same (and what windows / C-sharp incorrectly call "unicode encoding".)

[@Spartan322]

Technically ANSI Windows-1252 (what is functionally always used when Windows refers to "ANSI") is incompatible with UTF-8 as there are numerous UTF-8 bytes which are "reserved" (usually for multi-byte characters) which Windows-1252 uses as printable (single-byte) characters. If any of those bytes exist in the string when read as UTF-8 the encoding will break, which in a well developed system will merely produce a few erroneous characters. Now since Windows-1252 extends standard ASCII, most of the bytes in Windows-1252 will be read well in UTF-8, specifically all the common American used characters. The problem with the encoding only occurs when you have these non-ASCII characters either in Windows-1252 or in UTF-8, in which case if you try to read the either UTF-8 from Windows-1252 or Windows-1252 from UTF-8, you will get problems. Experienced this exact issue when dealing with a game that reads and writes Windows-1252 but which has resources written in UTF-8, causing all sorts of weird problems.

[@hellterminator]

@@laurentverweijen9195 UCS-2 can only represent ~6% of the reserved Unicode codepoints and ~69% of the ones already assigned , whereas UTF-16 can represent them all through surrogates. Don't get me wrong, UTF-16 is the worst Unicode encoding, but at least it *is* a Unicode encoding, unlike UCS-2.

[@rogehmarbi]

I almost completely forgot WordPad exists, despite it being more powerful than Notepad.

[@FlyTechVideos]

WordPad is too powerful for us mortals

[@npoaccount9154]

​@@FlyTechVideosyea, nothing can break it

[@masonboone4307]

Notepad is a decedent of wordpad

[@zUltra3D]

@@npoaccount9154 wasn't there a video about corrupting the heck out of windows 10 and everything was broken except for wordpad

[@Fidumo]

@@npoaccount9154 as soon as i read that, i tried breaking it. i ended up creating a document filled with a creepy smiley face image with the size of 2,306,727,936 bytes (that's 2.3 gb). i cant open it because it's too big, and i think i corrupted the file by closing wordpad while it was saving the file. i still dont think this counts as breaking it, i think its just the file thats broken.

[@y7o4ka]

Dude really digged into windows api's assembly to uncover a strange bug from 1994 Good job!

[@thevorkman_6551]

I think he not dive, he just wrote his own that works similar... Maybe, idk

[@aetimes2]

@@thevorkman_6551 He went into the code to figure out how it worked, and then wrote his own that worked the same

[@mstech-gamingandmore1827]

Don't forget the special... _sources_ ;)

[@thevorkman_6551]

@@mstech-gamingandmore1827 Yes)))

[@mskiptr]

XP source leak probably

[@jhgvvetyjj6589]

Aside from the statistical check, another heuristic uses the newline as a way to rule out Unicode since a word-aligned newline has the bytes 0D 0A, but U+0A0D is not assigned in Unicode. Also it apparently only detects based on the first 256 bytes or so, which might make the longest string challenge futile beyond that point.

[@FlyTechVideos]

oh wait, but that does mean that a non-word aligned newline could technically trigger it, right? should have spent more time researching nooooo

[@Renteks-]

just make this comment your string so you can win the meta award

[@ivirius.parody]

Ah yes. As a developer, I love to see people finding bugs in our lazy work

[@AlphaFruit-hx4cw]

I'm an programmer in my free time, and this is actually facts. 😂

[@dubl33_27]

@@AlphaFruit-hx4cw which Bush hid

[@nt-authority-system666]

you're spitting FACTS

[@UnixTMDev]

@@nt-authority-system666bush wasn't

[@Fasguy]

The amount of times i've seen this spewed about as an "easter egg" is nuts. Just like the "Bill Gates Sucks" "easter egg" in C64 BASIC.

[@chri-k]

what’s that?

[@brinleyhamer729]

yeah what’s that?

[@Dj_Theorema]

@@chri-k To keep It simple, C64 BASIC has a random number generator that, every time you turn on the machine, it always produces the same "random" sequence of floating-point numbers, all between 0 and 1. Using this "feature" someone wrote a small program (4 lines of code) that print the sentence "Bill Gates Sucks" on screen

[@egeakan7276]

God, when I tell you I thought Bush was a literal bush from nature and I couldn't figure it out until this day...

[@mucookul]

Same

[@Aeduo]

Sounds like it'd be related to homer simpson creeping out of that bush gif.

[@WindowsDrawer]

Same

[@sadpeperoni7508]

My wife's bush hid some facts. I'm catholic, so I discovered the truth only after the marriage

[@Liggliluff]

Yeah, for people not from USA, or not familiar with USA's presidents, the name "Bush" is likely going to refer to an actual bush instead.

[@Kiwifruit00]

4:19 for anyone wondering why fly wrote it as 0x75 0x42 when the hex editor shows 0x42 0x75 its because the file is encoded in "little endian", which means the last byte in the hex editor goes first when the computer is reading unicode. i dont know why the computer does this, nor am i an expert in these kinds of things but i just wanted to share in case someone wants to know

[@johndododoe1411]

Windows uses little endian because the x86 CPUs do so .

[@leap123_]

because x86 (which windows and every single os that supports x86 runs on) use little endian, so fuck intel i guess

[@RedstoneNguyen]

Write a simple program converting from integer to string and you will find out why little endian is a thing. Btw, the numbers we are writing everyday is big endian.

[@johndododoe1411]

@@RedstoneNguyen Both storage directions make perfect sense . Little endian for decimal digits is how Arabs write Arab numerals, big endian is how westerners write the same numbers with the same digits . Because computers gulp up entire binary numbers in one memory clock cycle it's entirely cultural for them too . The x86 and x80 CPU families belong to little endian design cultures, the 68000 and SPARC families belong to big endian design cultures . ARM and PowerPC hardware is bilingual in this matter .

[@RedstoneNguyen]

@@johndododoe1411 i didnt say anything about culture. My idea is, little endian is mathematically simpler to implement than big endian.

[@Blaineworld]

now i kind of want to know how the current unicode detection works

[@keltrm]

I tried disassembling it, but it seems to have been moved to the kernel (RtlIsTextUnicode)

[@Milennium1902]

It uses the 2 bytes FF and FE shown at 5:24. To make the glitch happen on modern Windows you put ÿþ into the beginning of a text file, then save it, and voila! You don't even need to input any special text after it.

[@lunafoxfire]

they probably do the sane standard thing and look for 0xFFFE at the start

[@chri-k]

@@Milennium1902**Icelandic obtains ÿ**

[@hellterminator]

@@lunafoxfire They did that back then, too, but that's not nearly enough. The _presence_ of the BOM _confirms_ a file is unicode. The _absence_ of the BOM _does not_ mean a file is _not_ Unicode. That is to say, if there is no BOM, you still have to check if it's Unicode.

[@SuperCaitball]

The "oracle" not guessing correctly on newlines might be due to differences in newline coding; given it's a Python script it may be using only LF line endings, but famously Windows always uses CRLF as its line endings.

[@boalbads]

"we are warned not to change it" "So let's change it" I love this channel.

[@2520WasTaken]

Have you heard about the Russian city: Seversk? It has a humid continental climate maintaining a low temperature and receiving 530mm precipitation every year. Through its presence, nuclear weapons have been assembled there and stored. One serious nuclear catastrophe would occur in 1993 because a container holding a dangerous and radioactive substance exploded. Character count: 362 Edit: I didn't actually test this on Windows XP notepad, but I used a script, and the script gave 7640 and 2542, and 7640 is just barely greater than 3*2542

[@AdachiVlogsFIN]

Saves And Does Not Corrupt On Windows XP Media Center Edition 2005.

[@moregirl4585]

@@AdachiVlogsFIN Confirmed on mine

[@Lolph_NG]

whats M69 doing here

[@Sypaka]

NO WAY LMAO.

[@fr4ctalz638]

tried it on my windows XP vm it worked

[@e_g..]

"This comment has the challenge shown for the longest strings that triggers the Windows glitch from the video you recorded. The video's bug shows that it's difficult for doing accidentally. Specially for the challenge proposed. I try using workarounds and the hardest one probably is the odd character words I have to usually put for those requirements. But sometimes I don't use odd length, which usually happens because the symbols are joined with the word. I created a small story for the text: The bus was not there for a bad reason, probably. Maybe someone found the reasons but I don't know? Which increases the words variety I can use for this! The current character counter is at 690 and I think I could add non-sense but that wouldn't get interesting enough. I'm a bit close here, 200 character distance. So, I could get another story using odd character count words only. There was a man named "e_g.". The challenge was waiting for him!! And so he broke the world record! FlyTech himself saw the text, and got amazed! The comment had too many characters and broke the record! I could not believe it! The story ended and thank you for reading this." this whole text has 1156 characters, which beats the previous record of 1016. the remaining balance was -399, which means you could add 2 more characters (following the rules from the challenge) without failing. a proof was made in a windows xp vm, and you can see the video proof in my channel or test it by yourself.

[@maelmauron7530]

The 00 padding on non-unicode characters explains the fact that in a lot of files that are not text, some strings have spaces between each character... I've had this question since 2017 XD

[@maelmauron7530]

By not text I mean archives, executables, etc.

[@chri-k]

but most modern applications ( one exception being Windows itself ) use UTF-8 and not UTF-16. UTF-8 is fully backwards-compatible with ASCII, so this may not be the only reason.

[@johndododoe1411]

It's not padding, it's the page number . Page 00 is mostly the same as Western ANSI code, there are about 7000 other pages to keep track of, including the ones with smiley faces .

[@Liggliluff]

All characters are Unicode characters. Some characters are also in ANSI, but they are also in Unicode.

[@hhkl3bhhksm466]

Your videos are always entertaining and informative. Keep it up!

[@proCaylak]

2:05 you don't have to disappoint them. there also was another Bush who was president between '89 and '93. In fact, he's the father of the Bush we all know and hate. I have no explanations for "~Flytech" though. 😅

[@mossadgynist]

He hid the fact that he personally assassinated JFK

[@aprilnya]

mans casually drops "my windows crashes when opening notepad" like HUH

[@FlyTechVideos]

kzbin.infoAtu7atNw-kw

[@zerotwoisreal]

what more can you expect from windows 11

[@UltraCenterHQ]

​I mean, it's an insider build. You would expect bugs

[@russelllukenbill]

When 7/11 (I say 7/11 because if I wrote the right date my comment won't post) happened, the very next day in school someone came in to school and showed a group of us in the computer lab if you typed 9 and then 11 into Word using wingdings, it was a plane flying into two buildings. Wingdings has been changed since then.

[@FlyTechVideos]

i'm pretty sure nothing stops you from posting 9/11 in this comment section

[@Xnoob545]

@@FlyTechVideos a different commenter said that "11:44 I can't do that as any link (sometimes even links to KZbin) cause the immediate deletion of the comment, meaning it's visible to its author until page reload, edits fail ("Unknown error") and reloading the page makes it disappear. Only the creator of a video can post links in their own comments section without needing to be worried" youtube likes censoring the comments

[@Biaanca5036]

I was never able to reproduce this bug when I was a little kid.. But well, my boxed copy of XP was Service Pack 2 so I guess that's a given :P But I also don't remember what OS I was using at the time either.

[@Cyberfishofant]

does that string really worl?

[@cathacker13]

for my whole life i thought bush hid the facts was an intentional easter egg so this was a very interesting video to me personally just because of that

[@selfSplintered]

Congratulations on being an official Wikipedia source! :D

[@MarkSir]

You didn't dismiss the conspiracy. Bush, the old one, was US president from 1989 to 1993

[@keiyakins]

the correct way to do this is, of course: 1. is there a BOM? If so, respect it. 2. try to decode it as UTF-8. If it worked, you're done, its UTF-8 (or us-ascii but thats a proper subset so whatever) 3. If you get here, complain to the user and make them figure it out. (if you're loading a document with more metadata you may of course use that too, I'm assuming plain text)

[@paradoxmo]

UTF-8 didn’t exist when IsTextUnicode() was written. The Unicode encoding in use was UTF-16 (based on the earlier UCS-2). So this is legacy code from the early days of Unicode that was never updated.

[@CattopyTheWeb]

Very interesting bug, Fly! Thanks for the video

[@Tocinos]

69th like

[@SsvbxxYT]

But what if...in Windows 3.5, Bush was referring to George *H.W.* Bush?

[@JustPyroYT]

Great Video! Very detailed explanation of the bug! :D

[@FlyTechVideos]

Thanks! 😀

[@ChloekabanOfficial]

The first time I found out about the "Bush hid the facts" bug, I thought the text was referring to a literal bush.

[@OrbitalCookie]

You can hide facts in a bush ... ... ..

[@ChloekabanOfficial]

@@OrbitalCookie Ah yes, my favourite hobby.

[@toydotgame]

This also explains what happens when you load Unicode-encoded files such as .lnk and .url in Notepad or Vim etc, and it displays the weird spaced out lettering as the program assumes ANSI plaintext. Cool!

[@623-x7b]

When I was a kid I used to open up exe files in a text editor. I thought programmers had to remember a lot of characters and had special keyboards.

[@DavidWonn]

NT4 Notepad runs on NT 3.51, though it will abruptly close at times. On these older versions, you can change Notepad's global font, and in some cases you may even be able to read the erroneous Unicode characters!

[@UltimatePerfection]

2:15 Bush Sr predates Clinton though as the president.

[@FlyTechVideos]

The Bush that "hid the facts" refers to the junior one, doesn't it? (Iraq war?)

[@UltimatePerfection]

@@FlyTechVideos Bush Sr. hid two facts though: That his son is a liar, and that he was a madman that almost started the WW3.

[@adrianv.v.4445]

Pairing a text-generating transformer with a minimizing function for the unicode check could be funny to see

[@Brahvim]

Exactly what I thought!

[@_AE_EA_]

Could be wrong on this but I think windows uses CRLF encoding so you would need to put /r/n into the oracle to replicate the notepad newline

[@FlyTechVideos]

I tried it with after the video, and the oracle says that it's censored while Notepad still doesn't break

[@PanoptesDreams]

All these years later, this explains why XP notepad was such a pee-pee about opening random text docs.

[@the-Gammaron]

"Legally not allowed to tell" Let me guess, reading leaked source code? Idk. You can't probably reply to this anyway. Or maybe it was a joke.

[@Milenakos]

the word "sources" is italic

[@the-Gammaron]

@@Milenakos Ahhhhhhh ok I'm dumb

[@maximiliano_sv]

@@the-Gammaron I still don't understand that about the sources, can someone explain it to me? is it some joke or something?

[@ruben_balea]

@@maximiliano_sv Basically Microsoft does not allow everybody to see the Windows source code or disassemble Windows to get the closest thing to the original source code, ChatGPT explains those things better than me: The source code of Windows is a set of instructions and programs written by Microsoft developers, forming the foundation of the operating system. While there are several reasons why an ordinary person cannot legally read the source code of Windows, here are some of the main ones: - Intellectual property rights: The source code of Windows is owned by Microsoft. The company invests significant resources, time, and effort in developing and improving their operating system. As a result, they have intellectual property rights over the code and can control who has access to it. This means they cannot allow just anyone to access and read their code without authorization. - Protection of trade secrets: The source code of Windows contains valuable and strategic information for Microsoft. Disclosing that code to unauthorized individuals could compromise the security, stability, and competitiveness of their operating system. Protecting their trade secrets is crucial for maintaining their market position and competitive advantage. - Risk of vulnerabilities and hacking: If the source code of Windows were available to the general public, it could be scrutinized by malicious individuals, including hackers and malware developers. This would increase the risk of discovering vulnerabilities and creating targeted attacks on the operating system, jeopardizing the security of millions of users. - License and end-user agreements: By using Windows, users agree to the terms and conditions set by Microsoft in their license and end-user agreement. These documents clearly state that users do not have the right to access, modify, or distribute the source code of the operating system. - In summary, the source code of Windows is owned by Microsoft and is protected by intellectual property rights and trade secrets. Its widespread access is not permitted to safeguard the security, competitiveness, and legal rights of Microsoft, as well as to prevent potential risks to users of the operating system. Disassembling parts of Windows to figure out the source code presents legal and practical challenges. Here are the reasons why an individual cannot easily disassemble Windows to obtain the source code: - Legal restrictions: Reverse engineering, which involves disassembling software to understand its underlying code, is subject to legal restrictions in many jurisdictions. Companies like Microsoft have legal protections in place to prevent unauthorized access, modification, and distribution of their software. Engaging in reverse engineering without proper authorization or a specific legal exception can infringe upon intellectual property laws. - Technical complexity: Disassembling software like Windows is a complex process that requires expertise in low-level programming languages, assembly code, and system architecture. It involves converting machine code (binary instructions) back into human-readable form. Understanding the disassembled code and piecing it together to reveal the original source code is a challenging and time-consuming task that requires significant skill and knowledge. - Incomplete information: Disassembling Windows and examining the disassembled code may provide insights into specific functions or algorithms, but it does not yield the complete source code. The source code of a complex software system like Windows consists of millions of lines of code, libraries, and dependencies, which cannot be fully reconstructed through disassembly alone. - Trade secrets and obfuscation: Software developers often use techniques such as code obfuscation to make disassembly and reverse engineering more difficult. Obfuscation intentionally complicates the disassembled code by adding irrelevant instructions, removing meaningful variable names, or applying other transformations. These measures aim to protect trade secrets and intellectual property by making it harder for unauthorized individuals to understand and reproduce the original source code. - In summary, legal restrictions, technical complexity, incomplete information, and code obfuscation make it challenging and legally risky for an individual to disassemble parts of Windows and obtain the complete source code. Reverse engineering is a highly specialized field that requires expertise, and engaging in it without proper authorization can have legal consequences.

[@safwan6363]

Thank you lol

[@kleinesfilmroellchen]

Even though Windows called it "Unicode", the less confusable and more accurate name is UTF-16.

[@VukAndrijanic]

Nice video keep it up! Will you upload more creepypasta videos like you did before?

[@FlyTechVideos]

Thank you! I only ever uploaded 2 of them, and no, I am not planning to continue them as I came to dislike even the 2 videos that I already made.

[@lior_haddad]

encoder? I hardly know 'er!

[@pyromancy8439]

I have moved to Linux completely a long time ago, and every time I stumble upon Windows, I seriously don't understand how can the most popular desktop operating system STILL have significant issues with encoding. Today I was on a video call with my coworker and had the pleasure of witnessing a modern (2021 version) app with cyrillic text display umlauts and diacritics instead of actual text on English-configured Windows., Look, mum, we've had Unicode for 32 years now! P.S. yes, I know Windows uses UTF-16, I refer to UTF-8, which is used practically everywhere on the web.

[@ZipplyZane]

I was hoping you'd then follow with how they fixed the bug. How does Notepad in Windows 7 detect Unicode?

[@chri-k]

by checking if the file starts with 0xFFFE

[@fgregerfeaxcwfeffece]

Microsoft is well known to preserve traditional bugs. Even the Win10 installer still could not select a partition to install.

[@Sypaka]

please explain in detail.

[@liniarc]

So I ran a few tests and found the exact letters and words don't influence notepad's unicode detection algorithm too significantly. By far the biggest factor is the location of the space and punctuation symbols. If the space and punctuation symbols occur primarily at an odd index, then the unicode detection algorithm can get a large bonus towards the odd/lower bytes since the ascii distances between the punctuation symbols and lowercase letters greatly exceeds the ascii distances between any two lowercase letters. This means that if you use words with an odd number of letters, most space symbols end up on the odd indices. However, writing sentences using exclusively odd number of letters for all the words isn't easy. You can therefore sometimes use pairs of even words for a nicer structure. By using these tips, you can write quite lengthy sentences which sound almost completely natural without having to recalculate a new score every other character. PS. This whole comment would get censored by notepad

[@ZephyrysBaum]

Wow

[@liniarc]

Character count: 1016 3 * Higher Diff - Lower: -85 Tested successfully on Windows 2000 pro With the heuristic outlined, it's fairly easy to make arbitrarily long strings, especially if you aren't overly concerned about clarity, word flow, or long term sentence structures. I wrote the comment without significant scripting/code assistance (only checking the isUnicode value every sentence or so).

[@ZephyrysBaum]

@@liniarc I’ll try making one I think

[@skyegibbs4955]

4:16 Saying that everything in "Unicode encoding" is 2 bytes is a bit misleading. This applies only to the implementation of Unicode used in very old versions of Windows (UCS-2), and does not apply to any modern, variable-width Unicode encoding. Notepad received support for UTF-8 and UTF-16 with Windows 7.

[@skyegibbs4955]

en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows en.wikipedia.org/wiki/Windows_Notepad

[@LeonAlkoholik67]

Kinda expected that it's just another encoding issue. Even nowadays Notepad still has similar encoding issues like when you write a script and it contains some special characters in it... and then you realize your script is broken. Aside that you should never use Notepad for scripting anyway...

[@intron9]

"in unicode encoding, each character is 2 bytes" Not exactly,but close enough explanation...

[@Gameplayer55055]

UTF-16. Then they wanted some nice 🥵emojis and even ♔chess. Then UTF-32 appeared. yet UTF-8 is variable length which saves the space but not the nervecells of c++ devs

[@Mnnvint]

In the old Windows World's favorite unicode encoding. Which they got stuck with, even though it was a bad idea, because they were too eager to use unicode and more sensible unicode encodings hadn't caught on yet.

[@jhgvvetyjj6589]

@@Gameplayer55055 ♔ fits in 16-bit character though

[@cl00e9ment]

It looks like they encoded the code points directly. They did not use UTF-8 encoding or else ASCII characters would be only one byte, and they did not use UTF-16 encoding either because UTF-16 is not padded with NULL bytes. In other words, they succeed to mess up their Unicode implementation and invent a new encoding while Unicode was supposed to unify everything. Oh the irony...

[@jhgvvetyjj6589]

@@cl00e9ment All ASCII characters do have null byte in high byte when represented in 16-bit integer though. 0x20 in 8-bit becomes 0x0020 in 16-bit, which becomes 0x20 0x00 in little endian, which is the correct little endian representation of space in UTF-16.

[@joveaaron-real]

If Windows is smart enough to remove the first two unicode bytes (0xFF, 0xFE), why the hell didn't they use it to detect unicode aswell?

[@FlyTechVideos]

They did - 0xFF 0xFE was consistently recognized as Unicode. The problem is that they assumed that text without this prefix can be Unicode as well, and they used the presented heuristic to guess

[@joveaaron-real]

@@FlyTechVideos sounds like somebody in Microsoft didn't read the documentation all the way 🤣

[@jhgvvetyjj6589]

Microsoft detects Unicode even without the byte order mark by design since other programs and/or platforms may save text files like that.

[@ThatRandomToast]

2:58 This variant of BSOD is caused by using VMware SVGA 3D

[@quantumelle]

How about this one? He erected later a great monastery in which he lived forty years and had eight hundred and eight followers--they bound him tightly and carried him between them on their shoulders (-9)

[@stephaniethebatter7975]

Bush didn't hide the facts, but Windows certainly did. (not part of the challenge, just a joke)

[@CrushedAsian255]

Before watching , let me guess is it some kind of Unicode auto detect mode bug

[@rainbowskeppy5292]

correct

[@Liggliluff]

(4:15) Incorrect, Unicode itself doesn't require 2 bytes per character. Unicode is just a list of character. It depends on what encoding you use. UTF-16 is what Windows uses, which requires 2 or 4 bytes per character, and there's also UTF-32 which requires 4 bytes per character, as well as UTF-8 which is a variable number of bytes per character. (5:25) FF FE doesn't mark it as Unicode exactly, it does, but it also marks it as UTF-16, which is why it's 2 bytes per character. B is 42 00 as you show, but FE FF is also UTF-16 but reversed, where B is 00 42 instead. EF BB BF is UTF-8, for example. (6:40) Since UTF-8 (Unicode) is variable length, why can't this be an odd length? Characters can be variable length from 1 to 6.

[@dedr4m]

Hmm, that's why this never got fixed! Somebody reported "this app can break" and the debugging team didn't get the message.

[@jetseverschuren]

And that's why everybody just uses UTF-8 nowadays

[@jhgvvetyjj6589]

Which is easily mixed up with all the other 8-bit encodings leading to more glitches!

[@jetseverschuren]

@@jhgvvetyjj6589 for text, really only plain ASCII or UTF-8 is used, at least in sensible systems

[@Foxy_AR]

9:45 if you were to spam a lot of these blocks, could you write secret messages?

[@FlyTechVideos]

If you consider mojibake secret, then yes

[@nothing-lo8lh]

@@FlyTechVideos Well no because it's fixed in newer versions of windows.

[@Gameplayer55055]

I live in ukraine and i've seen many university pages full of them (idiot devs, no ) After making essays in vscode i can see some mysterious п»ї too. its shit from BOM as i know

[@aurastrike]

@@nothing-lo8lh You can cause it by adding ÿþ to the start of a .txt file

[@DacroyleYT]

Dang, I thought the "Bush hid the facts" thing was an easter egg until I saw this

[@FlyTechVideos]

flies.sh/discord

[@batorerdyniev9805]

No

[@busymaan]

No

[@r0fael_programmer]

Fly!

[@busymaan]

@@r0fael_programmer and the no chain is already broken

[@thepikachugamer]

@@busymaan cause its kind of cringe

[@KnakuanaRka]

I’m guessing as for why it detected Unicode like this, the idea is that legitimate Unicode text will mostly contain symbols from a single alphabet, which are close together and thus will have high bytes that are close, so the sum of differences in high bytes will be low. If it isn’t, the high bytes (made of every second letter) will be all over the place, making the sum of differences higher. Would be good to have an explanation like that in the video.

[@FlyTechVideos]

You're right... thanks for the suggestion

[@mfaizsyahmi]

I'd imagine Raymond Chen would write about this embarrassing algorithm implementation in his blog soon.

[@FlyTechVideos]

He already did: devblogs.microsoft.com/oldnewthing/20070417-00/?p=27223

[@jiaan100]

Did you know conspiracy and conspiracy theory were words in 16th and 17th century, but conspiracist and conspiracy theorist weren't words until the 60s and 70s.

[@Lopoi]

Can videos be used as citation? I thought it had to be academic articles or something like that

[@FlyTechVideos]

Not sure if they _can_ , but I think I've seen some. Don't take my word for it though

[@rizkyadiyanto7922]

even blog posts can.

[@shepardpower]

I think so

[@wiger_]

generally self-published content is not supposed to be used as a reliable source for a citation, but in this case, i guess it could be used as a showcase of a behavior mentioned in the article

[@renerpho]

@@FlyTechVideos I've been working on that exact same question for a different video lately (a Karl Jobst documentary). The short version is that the video can not be used unless it is published by a reliable source. Since KZbin videos are self-published, they don't count. An exception can be made if the person who posts the video is considered a subject matter expert. We've discussed that for Karl Jobst, but determined he doesn't qualify. For it to work, Jobst would have to have published articles about his work in trusted sources, outside of KZbin, and he hasn't done so. What that means for your video: Have you published journal articles about PC bugs, under your name? (Just being cited by them is not enough.) If the answer is "yes" then that's great! Please give us a link to that. With some luck, that will make you pass as a subject matter expert, and THEN we can start thinking about citing your KZbin video. P.S. For Jobst's video, we could avoid citing the video in the end. Jobst presented all his sources in the video. References to reliable sources that demonstrate that his conclusions were correct. Can you share such a source for your topic? In that case, we can start working on the Wikipedia article as well.

[@mfessi]

21x johnsparks still does the trick. Source: "Windows Notepad - an Old Problem Surfaces Again"

[@netkv]

i completelly forgot some insane systems dont use utf8

[@Mnnvint]

The windows world started using unicode before utf8 was invented. The Java world too. Sometimes it pays to be slow (although I remember switching my old redhat/mandrake systems over to default to utf8 was not fun either).

[@johndododoe1411]

Unicode was created in the late 1980s. Microsoft and Java chose the early 16 bit Unicode and then had to use UTF-16 to encode the next few thousand pages . Then someone decided that any characters not handled by UTF-16 should be banned from all other encodings .

[@andyhu9542]

I can see where this comes from: if someone types in Chinese (or other Unicode-only) language, the characters will be very close to each other in the encoding space. Therefore, the most significant byte would be very close to each other, while the least significant byte would essentially be random.

[@jacky7204]

While if someone types with just a few Unicode characters thrown in, the ASCII values will produce the null bytes seen at 5:11, which also won't add to the most-significant-byte-difference counter.

[@kaninchengaming-inactive-6529]

Least broken microsoft product:

[@rainbowskeppy5292]

💀

[@IvyANguyen]

Learnt something new in your video: Mojibake! I never knew this phenomenon had a specific name. I recall seeing it a lot in 1998 to 2001 on WebTV here in the US when viewing Japanese, Chinese, & Korean web pages. (Really, any pages that didn't use the Latin alphabet.)

[@Neubulae]

Try "联通", iirc this single word also caused issue therefore similar "rumors", or rather, memes, arose amongst Chinese communities about how China Unicom had vendetta with Microsoft whatsoever.

[@alex15095]

I modified llamacpp so that it would only sample tokens that followed this rule, and asked it to generate a story about cats. The result is horrible (the sampling could probably be improved to bias rather than outright rejecting tokens) but it seems to break in notepad! Once there was a cat named Tabibabia who lived a happy and content existence under the watchof her owner’s grand parents who lived a comfortable and peace ful lilting langu age where nothing could disturb their daily routine neither did any worry bes eckon their minds for there was nothing but happiness and joy all round and every thing tumbled along

[@FlyTechVideos]

You added additional spaces inside words, that's cheating! (Also, you need punctuation) nonetheless a cool attempt at scripting :)

[@Dontlokhere]

Bush hid the facts 畋样凭摩琠映捡獴 Picking up mongoose

[@hackdesigner]

Visual C++ 2008 Express Edition? I see a man of culture!

[@cmyk8964]

But why was that “low > 3*high” heuristic chosen?

[@FlyTechVideos]

A unicode string with "simple" characters usually has a lot of null bytes, e.g. 42 00 43 00 44 00 ... and the heuristic is engineered to detect exactly this. As we can see, this leads to false positives

[@TheGodOfAllThatWas]

​@@FlyTechVideos Is that logic true across most languages or is this an English bias? Are there any countries where it's not true?

[@DrKoneko]

Whatever you say, fed.

[@barfooguy]

I'm just wondering, how can the windows notepad crash windows?

[@RandomGeometryDashStuff]

buggy gpu driver?

[@sebastianx708pl]

When I used RTM version of Win 11 on VMware after release, while using paint crashed into BSOD in few seconds so could be a VM problem

[@BlueFlame_00]

10gb+ files

[@johndododoe1411]

Win11 notepad is very buggy, it can't even display actual text files .

[@RandomGeometryDashStuff]

@@johndododoe1411yes, microsoft made notepad slower than notepad++ same with cmd.exe (old one still exists as conhost.exe)

[@RabbitEarsCh]

Real unicode detection is pretty expensive, so this NT bug got to live for quite a long time...

[@cs127]

great video. your explanation was perfect!

[@UltraCenterHQ]

Tinfoil hats are going wild rn

[@coolmannder]

you can't make a video showing something in windows xp without dreamscape and bandicam

[@FlyTechVideos]

dreamscape is content-id protected and because i am not desiring to donate all my income to 007 sound system i would rather not but yes

[@CattopyTheWeb]

lol

[@MMMMMMarco]

In unicode, a character isn't 2 bytes. Unicode itself is not an encoding, just a standard. UTF-8 is unicode where each character uses 1 byte or more, UTF-16 uses 2 bytes or more and UTF-32 always uses 4 bytes for each character.

[@CBFNetworksArchive]

Petition to make his videos way frequent: 👇

[@thepikachugamer]

That will overwork him

[@CBFNetworksArchive]

@@thepikachugamer Why? He is making videos every 4 months! Is it too rare?

[@Alexander-oh8ry]

​@@CBFNetworksArchive Petition for you to 1. not decide that for another person and 2. to do your math

[@CBFNetworksArchive]

@@Alexander-oh8ry Petition to you to 1. do not send me messages if you're a alexander (yes, it's on purpose) and 2. not to make me do things that I shouldn't do today or after 2 weeks

[@Alexander-oh8ry]

@@CBFNetworksArchive dam somebody sounds insulted because they cant force a youtuber to do more videos and is called out for it (and because they cant do math)

[@GrizlikD]

FlyTech, after he held a Microsoft employee in his basement for the past year: _"I am legally not allowed to tell you how I figured out. Let's say, I consulted some trustworthy sources for this."_

[@SirAU]

Yes.

[@SkigBiggler]

More likely he used he leaked XP kernel source

[@SirAU]

Wait, is that my uncle?

[@Experiment626.Stitch]

Ich mag den Glitch: Bush hid the Facts-Glitch

[@christopherg2347]

I can only point to this 2 decades old article: "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)": *The Single Most Important Fact About Encodings* If you completely forget everything I just explained, please remember one extremely important fact. It does not make sense to have a string without knowing what encoding it uses. You can no longer stick your head in the sand and pretend that “plain” text is ASCII. *_There Ain’t No Such Thing As Plain Text._* If you have a string, in memory, in a file, or in an email message, you have to know what encoding it is in or you cannot interpret it or display it to users correctly.

[@Tigrou7777]

It would be interesting to find out how they fixed it in newer versions of Windows (if that encoding is still relevant). What is a quick fix or did they choose a totally different method ?

[@Sypaka]

doesnt work on Windows 10... but on notepad2.

[@Lampe2020]

11:44 I can't do that as any link (sometimes even links to KZbin) cause the immediate deletion of the comment, meaning it's visible to its author until page reload, edits fail ("Unknown error") and reloading the page makes it disappear. Only the creator of a video can post links in their own comments section without needing to be worried.

[@JustAPersonWhoComments]

Plot twist: Turns out, the entire video was just a secret message from aliens, and they were trying to communicate in their own funky beatbox language

[@harrytsang1501]

I am pretty sure Windows XP notepad messed up this half the time if you are using CJK languages. There's a reason we wait a minute for word to load since it is slightly less prone to this but still not immune. The widespread adoption of emoji really made western developers aware of unicode and text encoding problem is rarely a problem anymore

[@Diamond_Tiara]

>windows 11 >open notepad >crash how dead windows can be, and the users ignore that.

[@Coastal_Cruzer]

you sound like the least pretentious ubuntu user

[@Diamond_Tiara]

@@Coastal_Cruzer Debian, thank you.

[@hundvd_7]

2:32 The most pathetic thing about Windows is that they used to have a very actionable error message, and they got rid of it. That's exactly the trajectory they've been taking, _especially_ in Win10 days

[@kuremaClaimer]

Windows should store metadata including encodings in Alternate Data Stream, maybe. Or all plain text file should use first some bytes to store encodings like Byte Order Mark of Unicode. It's too late though. Detecting encoding is generally done statistically therefore it's not most stable in the world. Some Asian country have editor app for the region because it handles these encodings used in the region better. For example in Japan, Shift-JIS, EUC-JP, JIS and UTF-8 is widely used and you can ignore others used in other region, you can write specific program to detect which is used in the text. Now that everybody uses UTF-8, it's a little old story.

[@--zero]

at the start of the video i had figured it somehow wrote out a unicode byte order mark with ASCII text, i am happy i was wrong about that. however, that IsTextUnicode function feels kind of painful :(

[@edivaldobetta4437]

why do i love the style of the windows and why have i never seen it before

[@SilaDrenja]

wait, but still all ASCII characters should be valid unicode characters. At least for the UTF-8 encoding.

[@zizhengguo3888]

I think I know why newline prevents the bug: it's because newline is actually two characters in windows notepad, which makes the total length odd instead of even.

[@FlyTechVideos]

No, I tried matching it up to newlines being 2 bytes and it still failed to trigger

[@MatthewTehSpartanUpdated]

1:50 This is truth that WordPad is un-breakable.

[@human.earthling]

Nice music at 1:00

[@ghost_ship_supreme]

This would be an interesting game mechanic for secret codes