I Only Test In Production

SPONSORED BY POSTHOG
Unit tests, integration tests, staging tests, what about testing in production? Great question. I think we should do it a lot more
Thank you PostHog for sponsoring the video!!! Link to blog post
posthog.com/pr...
Check out my Twitch, Twitter, Discord more at t3.gg
S/O Ph4se0n3 for the awesome edit 🙏

[@TomNook.]

These thumbnails are way better than those fake tweet ones

[@PatrikTheDev]

Agreed, the fake tweets work but this feels way more legitimate

[@zyzlol]

I skip the videos with fake tweet thumbnails

[@Sammysapphira]

​@@zyzlolme too

[@ridass.7137]

crybabies, here they come 👶

[@Quephara]

Theo is the reason I got an extension to change thumbnails on KZbin. He has good videos but the thumbnails were that bad

[@ottomanherox]

Just test in production so your users can be the QA team, save costs!

[@jbkerns]

Works for Microsoft and others

[@marktiempo466]

works for many AAA games

[@rapzid3536]

Shift left.

[@leetdavid]

To quote zack freedman: in practice, the difference between theory and practice is greater in practice than in theory

[@MrDadidou]

The difference between theory and practice is that theory does not work in practice

[@andrewgora5446]

If in theory, the difference between theory and practice is less in theory than in practice; but in practice, the difference between theory and practice is greater in practice than in theory; then in theory, is the difference between theory and practice actually greater in theory than in practice?

[@samcalder6946]

Every app has a Test environment. Some of them also have a separate Production environment.

[@matteusan]

time to push to prod on save

[@codeChuck]

Best strat ever!

[@thomasgeorge5261]

I work at a payments company and often you don't get a second chance - if a code change means money starts getting routed to the wrong account you can't just revert the PR and laugh it off. Tests are not optional in this domain

[@joelv4495]

Yep. When money gets screwed up, problems can become existential in a hurry.

[@StefanErwinBaumer]

They're not advocating for optional tests, they're advocating for the fact that at some point tests will help prevent fewer errors than better tracking in production. You very likely already have a trivial yet immensely important tracking mechanism in your system: logging transactions, which is essential for finding faults and reverting damages - in fact I'm fairly certain that not having this measure will land people in the financial world in jail. A less obvious but very actionable point is canary or shadow launching, which are both techniques for finding faults in production: canaries are about deploying a change to a small share of users to see if it works, that way you can find faults earlier with less damage. The expected value tradeoff should be clear here: at some point writing more tests will lead to more catastrophic transaction fuckups than not writing the last few tests and instead implementing a canary system to spot mistakes early. Remember that tests do not prove your code works (unless they check all inputs or are a formal proof), they just mean you didn't find a fault you expected shadow launching will deploy your new component alongside the existing software, with copies of the real data, or just no write access; and then every user request gets thrown at both the old system and the new system and you check if their outputs differ, hinting at a bug in at least one of the two systems this discussion really isn't about "reverting the PR and laughing it off", it's about figuring out how to minimize the damage of your *inevitable* money fuckup. and i sure as hell would rather notice that i forgot to cover an edge case with tests when timmy is spending his allowance than when microsoft is buying another game studio

[@thomasgeorge5261]

@@StefanErwinBaumer we do indeed do a lot of something like “shadow launches”, where we verify that a change would result in the same payments being made, but canaries isn’t really an option as most of the services (at least, where I work) don’t really operate on a request-response basis like a normal web app - a request will initiate a payment, but from there we have a bunch of async batch processes which ultimately result in money movements, and you can’t easily just enable a feature for one user. I acknowledge that the point of this video wasn’t about tests vs no tests, but I think there was an implication that monitoring and fast rollbacks were a reasonable alternative to very strong tests, which isn’t true in all domains

[@n4bb12]

Absolutely. But many projects are different and slow themselves down artificially because they don't realize they're not NASA.

[@05xpeter]

Our precommit hooks include linting(we have developers on Windows, Linux, Mac, VsCode, PyCharm, with and without autosave, so to have the linting enforced and the same is essential), checks that you are not commiting to the main branch, checks that you are not editing commited migration files. I think it is only a benefit.

[@PatrikTheDev]

I know it’s not always possible but I would recommend merging on the same editor or at least same local formatter (so probably Prettier with a common config, so that users can write however they want but the result will be consistent)

[@robertluong3024]

We use husky to prettier and lint things. It just auto-formats things before you do your PR. It runs fast so you won't have to worry. For commiting to main branch would making it protected help? because it would stop you from pushing to it right?

[@StefanErwinBaumer]

Those tests should be run on a server managing the repository, not on the devs' local machines, especially because hooks can be skipped. especially committing to the main branch should be something that is handled by your repository's permissions, not a hook. Do you really want to stop devs from making a backup of their work on a feature branch at the end of their work day because of a linter noncompliance?

[@PatrikTheDev]

@@StefanErwinBaumer prettier wouldn’t ever block you from committing unless what you committed was not valid code/syntax, then it would probably be valid to stop the commit Doing so for linting is a separate issue

[@CTimmerman]

​@@StefanErwinBaumerJust look at the warnings sometimes. Don't force others to.

[@JacobGorny]

Incremental failure is a beautiful thing in prod. Way better than catastrophic failure in prod which happens with overconfidence.

[@michaelslattery3050]

Good points. Disagree that shouldn't have a staging server. It can catch a lot of deployment issues. I also believe in a simple smoke test that hits all the integration points (login, send email, charge CC, upload/download with S3, other external rest APIs). It just makes sure integrations aren't broken, it doesn't test logic. This further makes the staging environment important. It might just be a /health page. You can also run the smoke test on prod immediately after deployment.

[@Tynach]

I'm honestly not sure how their "Test on a server with a copy of the production data" _isn't_ just a staging or testing server. In fact, none of the things they list seem like 'testing in production' to me, but instead a combination of staging/testing servers with old copies of production data, combined with standard monitoring. I guess A/B testing, but that's less about testing code and more about testing if users like a change or not. In other words, it's not the code you're testing, it's the users.

[@mainendra]

How do you make sure that we’re not leaking feature in production and feature flag works as expected? 😊

[@TheVoiceofTheProphetElizer]

I often told people at my university that production was the greatest test environment I was aware of. Rarely do you get a second chance to make a first impression, and testing in production solves none of these things, but boy is it fun!

[@tomatao.]

re pre-commit hooks... it's sooo easy to make a git alias that appends `--no-verify` for a convenient opt out of pre-commit hooks when developing WIP. On the other hand, having that local check before waiting for a bunch of CI to start running can save hours over the course of a day. The important thing is that the pre-commit hooks should take less than 10 seconds - ideally 1 or 2 seconds Here's a bunch for anyone interested: cn = commit --no-verify cmn = commit -m --no-verify can = commit -a --no-verify camn = commit -am --no-verify cian = commit --amend -C HEAD --no-verify psn = push --no-verify pfn = push --force-with-lease --no-verify pffn = push --no-verify

[@m4rt_]

Fun fact, there is a hidden setting on discord that the developers use to test new features, and variations on new features. If you find a way to activate it (they sometimes patch the ways people activate it) you can try out new features early. (It's called "experiments" btw)

[@boscodomingo]

About pre-commit hooks: I agree. However, pre-push hooks are perfectly fine. You don't push anywhere as often, and the only time where it matters what you've done with code is when you share it

[@rangerboi9715]

Staging? What’s that? Real men test in prod

[@n4bb12]

0:40 "randomly click around" is not a bad idea (maybe just do it before prod). I've seen projects write hordes of tests just to produce a 500 on first use which they missed because they never used the thing they were building. You'll learn a lot about the thing you're building by using it.

[@ElvenSpellmaker]

_"Let me know in my Pull Request that my code is wrong, don't prevent me from committing because you don't like the way I did my code"_ --Theo 2024 I support this wholeheartedly, I've been championing this idea for years.

[@BlazeShomida]

When it comes to pre commit hooks, what about something like biome that lints and formats in the less than a sec? Or using husky to do lint/format (w/ Biome or eslint+prettier) as a pre-push hook? This way you can run your linting/formatting locally which will in theory run faster than your CI before you push and you can commit as much as you want, without slowdown. Also this way you reduce needing to rewrite your local git history for a formatting/linting change and force push to remote.

[@ShellBryson]

Hah. How about a bit of both? Side project only has a single prod database, but I have local dev and prod envs for the front end code... 😅

[@codeChuck]

A dancing crew should train a lot and make 'test runs' before giving the final performance. Testing - like repetition to become 'almost perfect' - is essential to give a good performance to the people. If you are creating stuff and do not test it, it will break and will freak out the users, so IRL you should become a Sir or Lady 'TestALot', so your users do not experience any incovenient bugs with your product.

[@michaelm3691]

14:40 I see apple lawyers finally forced ikea to drop the i.

[@codeChuck]

Like the video before watching :) Becaise od thumbnail! Like it! As a joke goes 'The real men rest in production' :D

[@allie-ontheweb]

What's wrong with staging just being a snapshot of production? All the benefits of this approach, none of the "oh fuck I've deleted a user's entire account" Edit: 2:38 I really wouldn't call that testing in prod though. But maybe that's bc it's the only thing I have experience with

[@trollingdirty8910]

Use the word "Validate" when it comes to "Producton." Then the CAB and CTO are onboard ;😉

[@AHMEDTHAER2]

Size doesn't matter good take lmfao

[@usmanabdurehman97]

Theo to WWE, confirmed.

[@andrej-ota]

Everyone is testing in production, some of us just have a good fortune to have a separate system that approximates some parameters of production in addition to production. Whoever imagines they're not testing in production either has more services than customers or is fooling themselves. Seriously: Software in production will invariably eventually fail, not accounting for it and prepared beforehand for it is asking for trouble.

[@joshuamanlunas8862]

Would people risk testing in production for a massive Internet Banking monolith, where a "small fail" costs a minimum 6 figure in fines by the government and various regulatory entities?

[@briankarcher8338]

I cannot express in words how much I detest Husky. The first thing I do when opening a new project is to turn that crap off. It is literally the worst package in the history of programming.

[@eliastouil7686]

Sad cuz you didn’t say “peace nerds” :,(

[@listener8228]

Best case everything works. Worst case you get fired 💀

[@Tooo2-zz1nn]

never wrote a test, my code is perfect from init

[@StefanvonderKrone]

Hate precommit hooks as well! They‘re annoying. And most of them can be replicated in a better in CI

[@CTimmerman]

But then you get incremental lint/bug fixing commits, which might upset people who want a "clean" history.

[@StefanvonderKrone]

I‘d rather have the CI just inform me about issues and abort the build. It‘s my duty to fix those issues

[@rjmunt]

Tell me you have a small team and a product no one cares about.

[@gintoki_sakata__]

We all start somewhere

[@doc8527]

​@@gintoki_sakata__ Everyone all acts like they are the tech lead/CTO/founder of something who scales the architecture from hundred to million, or a team of several to hundred. Speaking from real anecdotes: 99% are just random dude joining a small/big team by luck where everything already had setup for them. All they did is just creating a new account, or adding cli commends (by copy/paste the existing example as guidance then twist around) for new features. But they all act like they setup the whole infra pipeline. Even for building a new team, most just don't have an idea how to manage people and possibly scale the number back after a year of money burning, and obviously big layoff. Larger companies just appears to have more free money to burn, longer time to figure out all the team problems.

[@Malix_Labs]

They test in prod because of real datasets I test un prod because I don't know what i'm doing We're not the same

[@TheBswan]

You guys are testing? I just hit deploy and grab a new ticket

[@codeChuck]

True :D

[@ScottMaday]

Pushing to prod? Theo looks like he's pushing out the biggest log know to man in the thumbnail

[@RealDatBear]

Please please please learn about git commit --no-verify

[@zaneearldufour]

Re pre-commit hooks: Theos opinion is only valid for J/TS development on this one, because in python and Go, the relevant formatters are basically never blockers (and pre-commit always canceling the commit on files changed is better behavior than husky silently changing files)

[@ivanmaglica264]

TISAX certification (a subset of ISO 27001 for automotive industry) requires that your development process has staging environment. If you told to the auditor that you only test in prod, he would close the notebook and say "See you in six months".

[@thebunsenburner]

We have a test server just for this, which we push to before prod. Helped us avoid a nasty bug at least once.

[@SJohnTrombley]

That quote about the difference between theory and practice is a paraphrase of Yogi Berra.

[@joelv4495]

One thing that wasnt covered is DATA. Often the application code depends on assumptions about the data structure beyond the raw db schema. Yes, you could use zod but what to do about legacy data that doesnt fit the current model?

[@KhizarKhan2001]

That thumbnail 😬

[@m4rt_]

I once had to test a UI update directly in prod... that was fun. (sarcasm) I was tasked with fixing a huge bug with the UI, and I could only test it in production (because of some data, etc, that was missing from the test environments... we could technically have done it in a test environment, but that would require setting up a lot of stuff that we didn't have time to set up), so I attempted to fix the bug, and I tested that the code should in work in theory, but I had to test the actual UI by pushing it to production. After debugging some bugs in the framework we were using, I finally got it working after a few tries.

[@nobilissimum_io]

You didn't test the peace nerds bit in the end on your dev environment

[@moosegoose1282]

i fking h8 husky too

[@bradweir3085]

We have to use mock data because of HIPPA compliancy.

[@codeChuck]

Oh, no! It is zombies :) They broke the internet and Theo couldn't say 'Peace, Nerds' at the end of the video!

[@devOnHoliday]

As a php dev I edit the production code in notepad

[@CTimmerman]

Cool, except for the paste near the Titanic, but that's what waivers are for.

[@Kane0123]

Where else would you test code other than your own machine?

[@MuhammadAhmad-ke5cl]

for size of business maybe think binance

[@jackofjs2243]

I'm too early for this 😅

[@ej3281]

post hog? okay

[@osbyrne]

hell yea

[@devpitch]

Only fools test in production, everyone else mirrors into pre-prod/uat or startups. It's true however, that certain validations cannot be performed in lower env. But it's not the same as "testing". Production validation should only involve making sure components are deployed and configured.

[@5h4ndt]

It really depends on what prod does and what the impact of the change is. I've been a system administrator for companies that are manufacturing. You don't want to ever risk fucking up the assembly lines. But our BW process? Fuck that, they can wait a day or two if push comes to shove, test in prod.