"a$$word" LITERALLY SAVED PayPal | Prime Reacts

Recorded live on twitch, GET IN
/ theprimeagen
Article link: max.levch.in/p...
By: Max Levchin
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
turso.tech/dee...

[@dromedda6810]

The guy that wrote this deserves a fucken award for that article, the storytelling, the twists, the characters, a$$word, everything was top tier

[@eyondev]

So, literally "It works on my machine"

[@NerdistRay]

That made me laugh out loud lol

[@hariacharya2534]

Yea literally, 😂

[@Ikxi]

im crying

[@JustinLietz]

a$$word

[@jsbiff78]

Maybe the TEST environment should be the same as the PROD environment, like, maybe don't test on Linux but deploy to Solaris/BSD/HP-UX/whatever *nix. Testing on Solaris would have found this issue fast.

[@kyay10]

Ik the math looks very complicated, but basically it uses the cool fact that a polymomial of degree N is uniquely defined by N+1 points. In other words, if you give me N + 1 points on an N polynomial, I can reconstruct the whole polynomial and evaluate it for *any* value I want. For instance, a line is uniquely defined by 2 points. Similary, there's only 1 unique quadratic that goes through any 3 points you choose. So what the secret sharing thing does is it gives all 8 people their own unique points on a quadratic function (degree 2 polynomial), and basically any 3 of them can then completely recreate the function and find the key (which is, by construction, f(0)) Edit: the original explanation in the article is good in the sense that it tells you exactly *how* to generate such a shared secret, but it doesn't explain well as to *why* it works

[@homelessrobot]

very cool

[@davidjohnston4240]

Yes. Pretty much exactly that. I've held a key share before (for a now decommissioned CA). In the form of a card (holding the actual key share) and a personal password for the card. Keep in mind the polynomials are extension fields of GF(2) so that the whole thing can be represented with bits because it's on a computer and bits are kind of handy.

[@hakooplayplay3212]

Oh...cool, now I see

[@stoneHeHenge]

This is a much better explanation

[@pauld9690]

Wait wouldn't cubic need 4 points?

[@demolazer]

Great article, what a writer that dude is. Even better having it read to me as a bedtime story.

[@rotteegher39]

Especially when you are Ukrainian. Literally me who stumbled upon this video before goin to sleep

[@NithinJune]

fr

[@trothwell]

Also came across it going to sleep 😴

[@ianbelletti6241]

More like a woman on crack telling a story. Too many irrelevant details and sidebars. Just get to the point of the story. I don't need to know your mother's father's brother's wife's maiden name for you to tell me this story.

[@gownerjones]

Nobody in the world would ever expect password inputs to be SECRETLY truncated. This is insane. Who programmed that?

[@musashimiyamoto9035]

This was a while ago, so fair enough.

[@DavidTretheweydtreth]

It is 100% insane but I have personally experienced it too many times to precisely enumerate

[@RanCham727]

It was a time when RAM was expensive

[@DavidTretheweydtreth]

@@RanCham727 SECRETLY was capitalized for a reason

[@batatanna]

3am at a darkened cubicle is never how you want to start a story ngl

[@robmorgan1214]

Unless... it's instructions on how to escape the backrooms!

[@AdrianBawn]

To answer the question at 14:55 "what would happen if 6 out of your 8 people were on the same plane" When you implement systems like this, you make sure that never happens. If you need to send more than 5 people from that group to the same place, at the same time, you send them via different airlines, trains, cars, whatever, spaced far enough apart that the chances of a crash involving all of them is essentially zero. If you are implementing a system THIS secure, chances are you have the budget to deal with this kind of invonvenience.

[@NickSteffen]

Yea, even outside of this most corporations have general limitations on the number of people who were allowed to ride on one plane. One company I worked at, it was 6 VPs and 25 normal staff.

[@lennykogginsofficial]

They shouldn't even physically be in the same office, they should be on different continents

[@RanCham727]

Shoot I worked for a now bankrupt and closed big box store in the mid west and they didn't let more than 1 executive or C suite, or 3 normal employees travel together.

[@jsbiff78]

I wonder how this system dealt with normal staff turnover? I guess, probably, as long as you have 3 people left, the system probably had code in place to recreate the master key from the 3 passwords, and then have 8 people enter new passwords to generate a new set of 'shards' that can recreate the master key.

[@timothycallahan7956]

There needs be a website dedicated to “bringing production down” stories. They hit you in the feels. SO HARD.

[@robinator18ps3]

Probably one of the best articles you've reacted to! Well written and a damn good story!

[@complexity5545]

This should be acted out as a skit and distributed amongst all computer science undergrad classes. Really entertaining. My bank did something similar. Unknown truncating is a problem. You can't read all of the manual.

[@jsbiff78]

Yeah, you can't read all of the manual, but you CAN test on the same operating system you are deploying to in PROD, on a non-prod system. The biggest fail in this story was that testing was done entirely on Linux and then recompiled for Solaris with Solaris's POSIX lib (which was different) and deployed to PROD with, clearly, zero testing on Solaris. This issue would have been found fast in testing if they had tested on Solaris, before it ever got anywhere near PROD data.

[@skarlock5257]

As soon as I saw the word "Solaris" in the article, I immediately began to suspect I would blame Solaris. I wasn't disappointed! 10/10 would read again.

[@SashaInTheCloud]

You have to use more than just people in a multikey encryption setup like this. You use things like a backup set of keys in separate lock boxes at banks in different countries, with two keys per lock box, and then another backup setup with copies of books at everyone's nana's houses, there's always a way around the plane crash problem!

[@sharpfang]

Silently trim the password to 8 characters. What an amazing security feature!

[@hs3369]

You'd be surprised that this problem also happened with Sony. But that's a story I'm never going to say again.

[@MMLauritsen]

Shamir secret sharing is unironically the coolest thing ever. I highly recommend reading the original paper 'How to share a secret', it's only 4 pages long!

[@unowenwasholo]

If there was ever a story that highlighted the importance of debugging skills. (Well, at least until the post-script, lol. Also the importance of always having a rollback plan whenever possible.) Being able to take a single working case and derive further understanding about the problem from the diff of that and the non-working has been so much of my programming career. “Why did _this_ work?” is often just as important as “Why isn’t that working?”

[@cericat]

Also test on all platforms you're intending to use in your deployment environment. It's precisely why I'll probably never launch anything with an Apple version, don't have nor want the hardware under my roof.

[@bimsherwood7006]

This is why 'availability' is one of the pillars of security, along with confidentiality and authenticity.

[@Lambda.Function]

I've learned through several horrible mistake stories like this that it's better to be a little insecure and make redundant backups until things are working than otherwise. It's saved me a few times when I've accidentally RIPd things and had a sigh of relief that I had backups.

[@Catterjeeo]

Well, a paper copy of a key hidden in a safe is not the least insecure

[@chri-k]

i mean, that is exactly what he did, he just forgot he did it

[@maxwellrobertson4831]

​@@chri-kIronically may have been more secure that way (idk if I'm using the words right) since then only one person knew the file existed and where it was, but they didn't know what was in the file. So no one trying to find said backup would have any luck since the person who knows what it is didn't remember it existed or where it was. (Hope I explained my thinking in a comprehensible way)

[@chri-k]

@@maxwellrobertson4831 They accidentally made 2fa. The person who knows what the file in the file does not know what's in it and the one who knows what's in it it does not what it is.

[@volbla]

So if paypal is using just a single password again, we can go back to beating it out of someone?

[@kreuner11]

Maybe in 2006 yea

[@TechBuddy_]

In 7 years since the creation of my account on KZbin this is the second video I ever liked. The article, the delivery and the emotion was just perfect ❤

[@ThePrimeTimeagen]

:)

[@allahnbirkulu6942]

what's the first one?

[@TechBuddy_]

@@allahnbirkulu6942 too embarrassed to share that 😂

[@JustATempest]

What was the first one?

[@TechBuddy_]

@@JustATempest NO!

[@superitgel1]

I want to see a movie of this. Great plot 😄

[@vaisakh_km]

😂 i am going quit programming and start learning animation just to make this a over dramatic animated movie

[@homelessrobot]

coming soon to a theater near you "PayPalia: Secret of the Lost a$$word"

[@alexhiatt3374]

would watch.

[@vborovikov]

there is a guy who narrates stories like this. I bet he's going to make a video out of it. channel name is Kevin Fang

[@AnirbanDas5000]

I want Christopher Nolan to direct this. Like Oppenheimer.

[@oderchannel426]

Oh man I agree with this so much, I totally watched this 27 minute video in 16 minutes and I understood all of it. I loved it when "a$$word" literally saved paypal!

[@maxlife459]

Solaris messed up big time back then: WTF were they doing truncating passwords!

[@benb8075]

Would have been fine if the program told you the pw was cut short. Silently accepting a system modified pw is pretty bad form, regardless of how cool, neat, or useful solaris devs thought it was.

[@TheArrowedKnee]

@@benb8075 Regardless it just sounds completely insane

[@Yorgarazgreece]

@@benb8075 that's still not good. there should be hard validation

[@taragnor]

@@benb8075 Well it's a C function that returns a char *, it has no way of notifying the user that it was truncated. It can basically either return a null pointer or it can return some string and that's it. Like most classic C style programming it puts all the responsibility on the person calling the function to be aware how it works.

[@ゾカリクゾ]

@@benb8075 EXACTLY. A single f-ing printf and that's it

[@Yupppi]

What a breath-taking story. Like the best adventure stories for kids, the dude had been smarter than himself at every turn possible, both in making sure it was safe and that he could not fuck it up. A bit of like reading one of those Artemis Fowl stories where the kid just has planned every possibility before and rehearsed the alternate paths.

[@holmybeer]

"Language interpolation" f**ing killed me

[@MNbenMN]

Here it had me thinking ZZTop and that shack outside "language"

[@cheaterman49]

8:39 I mean it _is_ an accent aigu, and I'm honestly impressed you managed to put a name on it :-)

[@Ataraxia_Atom]

This legitimately made me LOL, dude a$$word must have been such the meme at PayPal

[@razt3757]

This demands a movie, I would actually watch it. Great writing!

[@roberthentosh5635]

All you have to do is type slower and press the keys harder. 90% of the time, it works all the time.

[@Turalcar]

0:10 Adi Shamir is obvously S in RSA.The others are Ronald Rivest and Leonard Adleman.

[@exception05]

It's probably about method of Solaris stores pass phrases. One of the features of DES is that it uses keys of a fixed length - 56 bits, which corresponds to 7 characters (if you count 8 bits per character, taking into account that the 8th bit was often used for parity). As a result, even if the user enters a longer password, DES only processes the first 7 characters. In the context of storing passwords, this means that if a system uses DES to encrypt passwords, it will only honor the first 7-8 characters of the password, greatly reducing its security. SHA-1 and MD5 are hashing algorithms and do not have such a limit on the length of the input data. They generate a hash of a fixed length regardless of the length of the input message. This makes them more suitable for securely storing passwords as they do not limit password length and provide a higher level of security.

[@Delfigamer1]

PSA: do not use SHA-1 and MD5 for security. They are considered too weak for modern computers. Use SHA-2 with the hash size of no less than 256 bit. PSA 2: do not use a hash function on the password directly. Don't even use it with a salt. There are algorithms designed directly for the purpose of storing and using passwords securely, called "Key Derivation Functions". The one you should use by default in 2024 is PBKDF2 with a 6-to-7-digit "number of iterations". PSA 3: also, in general, "don't roll your own crypto", but also be aware of the X-Y Problem. E.g. when you build a site and want to let people register accounts in there - don't google "hash functions", don't even google "password storage" - google "user authentication" instead (or "how to verify the person is actually who they claim to be" in normal people's language). The result will be that, for an online service, it's better to not deal with passwords at all, and instead rely on OAuth-ing accounts from other services, like Google, Twitter, Github, etc. Then they can do all the security that's considered appropriate at the time (passwords, 2FA, retina scans, whatever else we will have to deal with in the 2070-s cyberpunk dystopia), and your site will just have most of this security just trickled down by delegation.

[@exception05]

@@Delfigamer1 Good advices, although my original comment was about the PayPal case that happened when MD5 and SHA-1 were pretty new.

[@arsenymun2028]

I love Stencil Law Men. My favourite Sci-fi

[@Gomace]

Rule #1 of coding: It doesn't work on the first try. Even if you check the syntax, double check the syntax, double check what it's supposed to do, and even used it before, there's always some number that is in the wrong place, one semicolon that's missing, one letter that's incorrect, a spelling mistake in a variable name, or it does the complete opposite.

[@harpyja-eagle]

Who would have thought, a bad password saving a company.

[@MeriaDuck]

Halfway in and commenting something you are probably going to say. This seems like a procedure you need to rehearse regularly. I once worked somewhere where the database had a master and slave setup and the slave taking over master role was tested every month.

[@mollistuff]

On the edge of my seat here. A real crypto-campfire tale

[@harpyja-eagle]

I once locked myself out of a remote windows server machine. I changed the password using cmd and didn't realize that my password used an escape character. When I tried to log back in my password didn't work. After a lot of confusion, removing the escape character solved the problem.

[@TheFlutterQueen]

does escape character refer to a character that escapes other characters or a character that needs to be escaped?

[@wizardscrollstudio]

That story brought a tear to my eye. All I remember is something something and a bad word.

[@monster2slayer]

companies i've worked for have explicit and enforced rules that make sure key people can not fly on the same plane

[@davidyoder5890]

This has been the best article so far. What a ride!

[@randomnobody660]

just want to quickly point out Adi Shamir is in fact the S of RSA

[@core36]

Murpys law is a thing. Always expect your little project to not work the first time you try it on the actual system it’s going to run on. Hey maybe don’t let a script overwrite critical files before you are absolutely certain that everything else works? The printed masterkey in the letter was a good call tho. Guy knew what he was doing, just got a little confused.

[@robmorgan1214]

Man, that was a whiteknuckle sphincter puckering read. I felt it in muh feelz.

[@jeffreybritton3338]

I loved this story and presentation. How did you not recognize SSS at the very end though. Shamir Secret Sharing.

[@htx80nerd]

Story about Paypal being wildly incompetent. This checks out.

[@porky1118]

2:07 Oh, are you also responsible for all the other bugs at Netflix? Like after watching advertisement, audio and video aren't in sync anymore. Or when watching on web, I first have to start playback before the "Back to main menu" arrow appears. And I don't know how to get the season and episode list, but sometimes it just appears when reopening a tab where I was watching a show.

[@kzalesak4]

To solve the people on a plane issue, we are actually implementing this in an organisation i work for, where you split the keys into physical copies, that are tamper-proofed, and then you hand them out to people to keep in a safe place of their keeping

[@abhatem]

What a roller coaster of an article 👏👏

[@noredine]

That story reminded me of online recipes where the author always tells you their life story

[@triplea657aaa]

This article is the kind of thing that made me get a Math degree.

[@bobDotJS]

Listening to this dramatic reading gave me nerd PTSD

[@danielschmider5069]

Funny coincidence how "Solaris" is also a sci-fi novel by Stanisław Lem

[@vray2904]

And Lem predicted a lot of stuff that happens right now in technology.

[@gosnooky]

"I'm not the a$$word" "Well, according to the state of New York, you ARE the a$$word"

[@Burgo361]

I really felt the stress of this situation this storytelling was amazing

[@jimdiroffii]

I once deployed a new package to a single node to test it. That update went to every single node instantly, slamming the entire network, and grinding operations to a halt. Luckily, the update was successful, and everything came back on its own. Some mistakes you will never make twice.

[@thehibbi]

Such a great article, and you reading it makes it even better!

[@fuzzy-02]

Aaannnnnnd saved, under dad stories for future dad meetups.

[@0xTristan]

YOO why did I just realize Max fuckin' Levchin wrote that lol 10/10

[@abz4852]

This has to be the best article ever. Literally could be a movie scene.

[@DmitriyKuzmenkov]

This article gave me some serious Silicon Valley (TV Series) vibes. A password Big Head would use...

[@zperk13]

14:40 Bus factor? Nah! Plane factor!

[@dennismuller1141]

there is also the term truck number

[@chinmaykabi]

me watching this at 3:41 AM

[@taylrthegreat]

Literally beautiful example of sometimes short passwords are cool

[@StrengthOfADragon13]

"What if 6 of your 8 are on a plane together" this is an eventuality that has to be considered, you can't have more than 5 of them in 1 place or unavailable at any given time

[@max_ishere]

Dude got RTFM'd hard

[@lashlarue7924]

This is pure nightmare fuel, but Prime reacting to it with the happy ending (rawr) makes it all worthwhile.

[@ragectl]

I remember having to look at Solaris being able to have long passwords and longer usernames. Totally wild the system is built to restrict everything to a length of eight characters

[@yarbarbar]

Lagrange interpolation is the basis of Reed-Solomon codes, so would be fairly common knowledge to people in computing at the time.

[@alexhiatt3374]

thank you for writing this great article prime

[@zbot2123]

We call the designated survivor problem a "bus factor" how many engineers on the same bus crash would result in business losses. Low bus factors are pretty dangerous

[@zzyzxyz5419]

When did youtube start to update the view counter live? I can see it moving up.

[@spidaweb-u8f]

Just tbc. What won me over the most in the video.... 'push-it' by Salt-N-Pepper scene setting. I can almost smell the room they were in from the 90's all the way back to present.

[@fuzzy-02]

This served me content of greater quality than a million novels

[@manuelschneider224]

Absolutely amazing article

[@emjizone]

10:27 😂😂 "No Haskell needed" : does it mean "It's not even real Math." or rather "Not even Haskell can save you." ?

[@tylercornett2022]

That was as entertaining as it was terrifying lol.

[@cericat]

7:58 Filk is a musical genre that mostly grew up in fandoms since the 50s, with much of the distribution in the 80s and 90s, so yeah a geek party is exactly where you'd expect to hear it. If you ever get around to reading Poul Anderson he actually wrote at least one piece of Filk as well according to his wife. 16:10 Cymeks are from Brian Herbert's follow up Dune books, they were humans turned into thinking machines. We're talking about pre Dune history here, the Butlerian Jihad. Abslutely nothing to do with the Tleilaxu, gholas or face dancers. Your chat was messing with you.

[@x-Mick-x]

Greatest bedtime story ever.

[@glennmorrow2755]

That’s gotta be one of the best stories ever! 😂😊

[@blackAngel88it]

Wait, what was the solution? Did they retry it with their passphrases, but only 8 of the first characters?

[@martin7462]

This article is an absolute fever dream

[@semasemasemasema]

In 3am you either having the the of your life or stare at the selling trying to sleep

[@TankorSmash]

This is a fantastic story. Loved the video

[@sneed1208]

6:20 Solaris is a pretty famous book and film

[@tmerb]

this needs to be a movie

[@sebastianp4023]

The title of the Article could have been as well "How to lose 10 years of your life expectancy in one night".

[@DUDA-__-]

Oh it thought about Shamir secret sharing for a key to my PW Database. I like the concept.

[@airkami]

Do knowledge gates make things more or less secure? What if for the 8 people you secretly chose 8 other active employees without their knowledge and those people could use a 2fa they didn’t set up to access the pass phrase they didn’t create and have no idea what it goes to and they send it to someone who doesn’t know what it is, but does know where and when to enter it and then once entered, someone who doesn’t know who they are or what they did can observe if the passwords worked or not? Assuming the security motto is “all of it and more is never enough”

[@liquidsnake6879]

He had the master passphrase written down but earlier he said his push had overwritten it, so it was useless anyways lol

[@gwentarinokripperinolkjdsf683]

No, not the passphrase, the key. that key would have been able to decrypt the database still

[@lezzbmm]

this is fkn amazing lmfaoo

[@ThePrimeTimeagen]

It's my favorite article of all time

[@daw5268]

bruh your reading of this was phenomenal

[@JGComments]

This guy made a real Schmess of things.

[@jamesm4957]

The real demons really appear at 3AM

[@tutacat]

Filk music is a musical movement among fans of science fiction and fantasy fandom and closely related activities

[@yaghiyahbrenner8902]

wow incredible journey. dramatic story very well articulated.

[@cluebcke]

You're telling me the people who work at PayPals don't refer to each other as Pay Pals?

[@disruptive_innovator]

overwrite backup successful. what do??

[@hypergraphic]

That is such an epic story!

[@pv2b]

Hey, that's cool, I have something in common with the protagonist of this story, in that my father also translated Stanislaw Lem (into Swedish).

[@GeorgeDicu-hs5yp]

classic security trough obscurity..

[@marble_wraith]

a$$word... welp now we know what i'm changing my wireless SSID to 😏