$5,500 A Month Saved From One Grafana Query

[@xevious4142]

$5k it's rookie numbers. I have a friend who saved their company multiple millions per year. It was such a valuable contribution to the company that they honored him with a "meets expectations" performance review.

[@benhedges2827]

Brutal

[@cariyaputta]

Performance review is such a scam.

[@LearnValkey]

Apparently, they were expecting him to cut costs. Keep in mind that SDEs in efficient organizations can bring 10-50x their salary in profit (TC times number between 10 and 50). Only software houses squeeze 2-3x revenue from SDEs.

[@xevious4142]

@@LearnValkey bro describing the surplus value of labor and alienation like it's nothing

[@xevious4142]

@@LearnValkey this is still pissing me off "No you're supposed to get screwed out of the tremendous value you bring to the organization, don't you get it?" We need a union. If you believe I shouldn't be getting a chunk of the value I'm creating if it's 50x my salary, you're literally a scab.

[@matheusgomes9343]

Actually Prometheus is the one who makes the query, and Grafana is the UI who shows the data that was collected

[@deadchannel8431]

🤓good to know

[@_julian]

Thought so too but explained 19:49, it's Grafana Loki

[@EmiNNsoNify]

Grafana Loki is for the unstructured logs which he made into a graph, Prometheus is for the metrics.

[@antoniop3080]

You must be fun at standups

[@ReluctantSpirit]

Prom is the store

[@dankprole7884]

I'm a data guy but these videos are teaching me a ton about the issues the devs face on my projects. Turns out we CAN just use Postgres for that 😂

[@gabrielwisniewski5857]

i dont get why postgres isnt the standard

[@oncedidactic]

the answer is always postgres and SQL, unless you have a REALLY good reason not to

[@QWERTIOX]

​@@oncedidactici don't like to send strings to say what i want to do with my data

[@oncedidactic]

@@QWERTIOX are you ok with api requests though?

[@ITechoManI]

@@QWERTIOX that's why prepared statements exist.

[@kahnfatman]

NodeJs is the trojan horse that introduces slow code that cloud providers invest in. Complexed logic, long running code, hard to manage-> use js -> „we“ take care of the rest -> pay later

[@thekwoka4707]

Most aren't using Node nowadays for most JS functions.

[@nexovec]

FinOps are literally the guys deciphering AWS bills for a living.

[@JacobSantosDev]

Do the come to the conclusion that the company could save multiple times of $100k a year by removing finops team and their salaries?

[@jarvisb.6013]

​@@JacobSantosDev Based lmao

[@user0K]

@@JacobSantosDev u can say the same with any other type of dev

[@JacobSantosDev]

@@user0K sorry? The reply was a take on an old joke. At a glance, I am not sure if you are playing off my joke or making a serious and wrong take. But is it wrong? I guess anyone can be replaced, it just depends on how much effort the business wants over the team's ability to reduce that effort.

[@NotMarkKnopfler]

"The journey begins on AWS ECS". I think we found the problem.

[@brands2131]

What's the problem with AWS ECS?

[@thelol1759]

@@brands2131gets used wrong a ton, like every other tool

[@geoffh4861]

@@brands2131 Nothing -- he's just being a k8s cargocultist

[@nicgeorge6126]

@@geoffh4861ECS definitely has some drawbacks.

[@sonilpro8526]

meanwhile, my Java application takes like 10 seconds to start without any code.

[@TankorSmash]

Loving these more regularly-timed, longer length uploads. Thanks guys!

[@ricky2629]

A go program compiles faster than Node starts. So you could make a ephemeral pod which compiles the code before serving the request and still see performance improvement. Having a performance-critical application in JS is just a fundamentally bad idea. Imagine how much more money that company could save if they hired competent developers instead of JS scriptkiddies.

[@TJackson736]

JS was a mistake.

[@cbaesemanai]

When you are running front end services like this that are handling data from potentially millions of clients you had better be using go. Not doing so is simply pissing away money and asking for performance issues.

[@spartanA01]

Or just don't run them in ephemeral pods... that seemed like a critical architectural design mistake. If you want FAAS, then use a real FAAS solution like Lambda which re-uses warmed functions so you don't pay the start-up cost for EVERY invocation.

[@TheTim466]

@@spartanA01 I mean they were previously on non-emphermal pods.

[@Mathias-gt6cs]

@@spartanA01This was the thing i was confused about. Do they really start up a pod, process one event and then kill the pod?

[@ChecklyHQ]

@@spartanA01one of the benefits of ephemeral pods was nice isolation of customer needs, and scaling was more fine grained.

[@mickolesmana5899]

"They fixed the problem even if they didn't use the standard method", Me remembering when I basically made a simplified data frame because I didn't know pandas exist

[@adaliszk]

While I would also avoid Node for mission critical things, it actually has a way to compile into a binary. Though its not as efficient as the vercel compiler was which were able to significantly tree-shake the node modules so you end up with image sizes under 100MB :)

[@keeganconnor3224]

"There's usually some tools to help you out" Joke's on you bud, I am the tool that helps me out.

[@johnyepthomi892]

Such a tool

[@cyrus01337]

Tool boy

[@ilrisotter]

I can't believe they started monitoring their costs as part of ops, did all this work, and didn't even think about reverting their move to faas.

[@bobster852]

At Prime's hourly rate, writing that empty golang just blew a few years of cost savings. :D

[@sila_v_malenkih_shagah293]

i don't understand the joke. can smb explain plz?

[@theepicgamer1196]

​@@sila_v_malenkih_shagah293 Programmers are expensive. Good programmers are even more expensive. The salary of a software engineer is generally more expensive than the cost to run the application, so reducing the cost of running the application may not actually save money.

[@fleal_ucm]

@@sila_v_malenkih_shagah293 Prime took way longer to finish the go program than the node one. He's comparing the savings from the improved startup speed of the (release) go program over the costs of having a developer with Prime's pay check spend the extra time writing the go program. A joke, of course, but a valid concern of dev speed vs runtime speed.

[@robertluong3024]

​@sila_v_malenkih_shagah293 prime earns big money and hiring him would make the costs the same

[@__Michu__]

@@sila_v_malenkih_shagah293 06:30

[@biddls]

Take a drink every time he highlights a full line and not 1 off each end

[@Winnetou17]

I think I'm drunk!

[@AloisMahdal]

I can only imagine how many bank-saving queries i could have done if only i was able to navigate that Grafana's weird attampt at a menu.

[@Thegetitguy]

How I saved 50k by being outsourced to India.

[@lordadamson]

25:55 about the genetic algorithm part, There already exists a really nice tool for C and C++ called the American Fuzzy Lop which does exactly that to general input. loved using it a few years ago and found really interesting bugs with it

[@gardnmi]

This dude reads millions of api requests and then suggests postgres as the direct target 😂.

[@ChecklyHQ]

Aggregating times is a good general practice but isn’t going to help much as we’re in an ephemeral pod. With the Grafana Loki logfmt method, I could use real LogQL expressions (similar to PromQL) to query the unstructured logs and treat is as a metric.

Who does run npm install in a pod during startup in production environment except its a ci/cd process. I'd suggest them to prebuilt the container image within ci/cd and use the prebuilt container image in production. This should also reduce the startup time.

[@mattdoestechstuff]

As someone who's workplace is deeply entangled with java, springboot and kubernetes the idea of 3 second load times being bad is wild considering most of our apps take anywhere from 25-70s to startup depending on the size and the amount of CPU assigned to the pod

[@QWERTIOX]

I heard that you can use graalvm to make java faster

[@mattdoestechstuff]

@@QWERTIOX yeah it's possible. But It does come with a bunch of catches that can extend the lead time to change as java reflection isn't fully supported which most JSON serializers/deserializers require. So you have to either attach a heap of extra config or run a heap of profiling last time I checked

[@henrymaddocks984]

This is one of your best videos for software development best practice.

[@craigtildesley1501]

In my first dev job (30 years ago) I was made to code on a PC that was slower than what our customers had to ensure that CPU bound code would always be performant

[@Sommyie]

I saved 8 hours of human hours a day by automating Grafana screenshots. Good times.

[@KangoV]

For my containers I have Java compiled with GraalVM to a native binary. The image then only has the app and glibc. That's it. Startup is instant.

[@georgehelyar]

You can do this with e.g. rust too very easily. You can even statically link libc and the dockerfile is just from scratch, copy single binary, entrypoint binary.

[@doc8527]

I love the last zod take, or runtime validation in general regrading the tradeoff in software development. While the runtime validation is always great to keep the correctness of input data structure for specific consumption. Depending on how complex, how large and how frequent the data (especially large json) you are going to check, it's not free. It takes time and can significant slow down your entire application experience due to "excessive" checks. Especially if you are doing any mission or cost critical development, you really need to think about where you want to do the validation before hands or simply let it fails somewhere in a harmless way, "Every spot need to be checked, but also non every spot needs to be checked" and "you always need to pay your decision somewhere".

[@morosis82]

Using a typed language can help, because it will save you from needing to do these validation checks within the logic. We use Zod only to check data coming from outside, like the request body, the dynamodb record, etc. Once it's in the program you don't use it to check shape for like an if-then, wrap it in an appropriate type or add an attribute that tells you when you check it inbound and then use that attribute/type from that point on.

[@thekwoka4707]

The validation can potentially just be ignored if the source is also a controlled thing. And at that join you could also not use JSON to save more time.

[@robpruzan7292]

U can throttle cpu on chrome devtools btw

[@Draggeta]

And networking as well.

[@mauricioprado6395]

@primagen, sure V8 is the javascript engine, but I am pretty sure they name it like that after the car engines.

[@ThePrimeTimeagen]

probably

[@morosis82]

kzbin.info/www/bejne/nojLfn6Xoaqcmbs

[@Olodus]

26:15 Has Prime tried to use a fuzzer together with his simulator and asserts? That sounds close to what he is talking about when he says genetic algorithm, but I am not sure. I just recently used LLVMs libFuzzer in a C project at work to do sort of property testing on our code, with asserts in the code and on the generated responses being wellformed. The libFuzzer looks at coverage when it generates its input corpus, meaning it is usually quite good at figuring out what to input to find those rarely used code paths.

[@georgehelyar]

It also just sounds like mutation testing, although I really don't think it's a substitute for writing good unit tests.

[@DryBones111]

Looks like people will do anything to avoid building something that can vertically scale.

[@JGComments]

And all it took was 5 engineers to save that $5K, right? Lol

[@lzszl]

Permanent savings? It's 66k a year

[@DryBones111]

Assuming an average pay of $70k in the team with 4 weeks holiday, 5 engineers working on this full-time for a week costs $7,292.

[@JGComments]

@@DryBones111 increase that by 30 percent for fringe benefits like health and PTO and you’d be right on.

[@wassafshahzad8618]

@@JGComments but they still save money. even if its costs them like 15k they make it back in 3 months

[@JGComments]

@@wassafshahzad8618 I never said they didn’t save money, I said it probably took 5 engineers

[@user0K]

"Node has no compiler". Node has bytecode compiler, though. Bytenode A minimalist bytecode compiler for Node.js. This tool truly compiles your JavaScript code into V8 bytecode, so that you can protect your source code. It can be used with Node.js, as well as Electron and NW.js (check examples/ directory).

[@Dom-zy1qy]

One time, i tried to use two different validation libraries. For my containers deployed on AWS, i used superstruct, which is significantly smaller than the one on my front end. I felt clever in doing this, but I've probably had to spend a total of 3-4ish hours combined figuring out the slight nuances between the libraries. The funny part is my app has like 4 MAU, and 1 of them is me, so I basically wasted all that time.

[@thewhitefalcon8539]

Did you learn? Learning is never wasted time

[@user-lg4le8xr4s]

So it was a big pile of JIT

[@mohammedalmahdiasad6832]

thank you for the video and the time you put

[@TheFreshMakerHD]

my work wouldn't let me use Go for our lambda functions, so I quit

[@keoghanwhimsically2268]

Spoiler: The “anti-pattern” bit was clickbait. It’s coming “in the second half of [the] article”.

[@ThePrimeTimeagen]

agreed

[@DaveRogersEsq]

"TDD is not great" - never a truer statement has been uttered

[@JeremyAndersonBoise]

As a dynamic interpreted language dork I found the “compile cache” comment funny as. TIme In The Saddle made me cynical and obliquely snide.

[@westwolf48]

@ThePrimeTime how do you perfectly and consistently select all but the first and last letter every time you highlight some text? At first I thought you were just missing slightly, but after watching for a while it seems like you're doing it every time which is quite a flex of your mouse wrangling skills

[@victorwidell9751]

React does not make you build state into the rendering. Use any of the state management libraries. Since you mention Elm, I'd recommend typescript-tea, which I use at work.

[@Shocker99]

JIT = just in time It applies to all industries - not just programming; and especially not just compiling

[@timlind3129]

I see these postmortems and often the findings are rather rudimentary, 101, simple, basic understanding of technical architecture and things a junior architect or developer should already know....

[@thekwoka4707]

the sub dependencies shouldn't have been a real issue, since they are likely all ^ ranged. But if you use pnpm, it's easy to force them :)

[@RamkrishanYT]

8:52 😂😂😂 bro is cooking

[@spartanA01]

So where was the anti-pattern? Using consistent module versions is not an anti-pattern :p

[@DenDodde]

Title of part 2 is anti-patterns, so i guess the author didn't get to that yet.

[@paulward8087]

Uh ditch the crap ... 1. AWS ECS 2. Javascript ... get yourself on Azure with Dotnet and all this goes away.

[@hanifarroisimukhlis5989]

It's interesting seeing the root cause. Meanwhile in Rust i can duplicate of a crate, often mandatory since some dependency depends on old version, without major impact. Well, maybe a bit of binary bloat and slow link time.

[@mattymattffs]

Meanwhile my company was arguing about 5k a year... Like that's a week's salary for most of our devs that use the feature that cost us 5k... Telemetry/monitoring basically

[@jamesrivettcarnac]

I love evented data and queues. CockroachDB in some use cases. Usually I just use postgres

[@paul.tinsley]

I got the ick when i saw it was JS not only on the server but as a one off function container BUT it makes some sense when you see what their product actually does. The users are able to write somewhat arbitrary JS to perform synthetic tests on a site. I'd likely just run a v8 instance in golang both for performance of startup as well as a barrier of protection between my code and any kind of javascript escape someone could dream up to impact my code. Either way I at least understand how they got where they are...

[@p_s1dharth]

Grafana mentioned let’s go 🎉

[@vinylSummer]

Bro grafana gets mentioned literally everywhere

[@WiseWeeabo]

UI testing based on screenshots is cool

[@Lorofol]

Would really love to know what Prime thinks about all these JS engines implementing AoT compilation of js into machine code.

[@eddyrose3254]

5,500 a month in savings from fixing the problem, but the team's combined monthly salary is 25,000.

[@morosis82]

5 months return on investment, and that's only at the applications current spend. If their traffic goes up could be less time for ROI.

[@lewishousten5455]

Synthetic monitoring tool is the new latest manager buzz term

[@treelibrarian7618]

so maybe I'm missing something because this seems so obvious to me, but if pod startup time is the issue, and each pod starts then takes a work item from the queue, then quits... why not just take another work item from the queue instead, and only quit when there aren't enough requests to keep all the running pods busy?

[@privacyvalued4134]

3 seconds is not an unreasonable startup time for NodeJS. Your puny 1-liner, does nothing node example took 0.03 seconds to start on your local hardware. On a server, you're sharing system resources, so that'll jump up to 0.2 seconds. And then throw in actual libraries that need to be loaded and parsed, poof now you have 3 seconds. Or more. I've seen NodeJS take up to 30 seconds to reach a stable state. It's a terrible ecosystem and no one should be using it.

[@adjbutler]

Elm mentioned! Lets go!

[@michaelbrauner]

Thank you for these wonderful videos. Can you tell me what program you are using for your painting?

[@noahklauck]

The free-hand one is GIMP (Gnu Image Manipulator) and the one with boxes, rectangles etc is Excalidraw

[@d3stinYwOw]

So much talk about picking compiled language, but gl & hf with getting colleagues to switch languages. Works for greenfield projects, not for existing ones.

[@morosis82]

Can work for existing, but needs to be done like a strangler fig pattern. Was in a team that was replacing a program written in COBOL with Java, millions of lines do not get replaced overnight.

[@d3stinYwOw]

@@morosis82 I know it's doable, but I think in your situation you had approvals, willingness from team lead and colleagues etc. That's the hardest part of it.

[@oleksrow]

Also, minor version measured in hundreds is terrifying

[@thekwoka4707]

300ms is a long startup time even with node...

[@thekwoka4707]

could have made it faster with ESM. Since then it could be loading, parsing, and running JS at the same time. while require it will load one file, parse it, run it, hit a require, now go load that, parse it, run it, hit a require...

[@orderandchaos_at_work]

FinOps is what Flipper does.

[@Dazza_Doo]

I'm watching ADS to listen about more Javascript.

[@TeDynef]

who the fuck builds monitoring systems that cost that much in the first place??????

[@greyshopleskin2315]

People using node at scale xd

[@TeDynef]

@@greyshopleskin2315 is monitoring node more expensive than running these node nodes?

[@johnyepthomi892]

@@greyshopleskin2315the time they saved using node comes back to haunt them later when they’ve acquired market but losing on perf. This is exactly what’s going on. It’s like they weren’t aware of the downsides going in.

[@ZeroUm_]

You would be impressed by how much Splunk charges per GB indexed.

[@AloisMahdal]

monitor lizards from space? hall monitors? i don't know, just guessing...

[@cbbcbb6803]

What kind of computer does Grafana run on?

[@autohmae]

5:30 I think Java is still slower to start than nodejs though. This is the reason Javascript won in the browser in the first battle. And obviously Java had the compile step as well.

[@autohmae]

16:48 this is why I liked statsd it's just a single syscall for 1 UDP packet and the monitoring system does the aggregation. Pretty certain of the slow ways, this is probably the fastest.

[@autohmae]

It's funny, when you see these things, often we end up where we came from, in this case logfmt came from Heroku. Like a bunch of other things as well.

[@ib_concept]

why the zee shell $?

[@d_6963]

2:50 It is Postgress underneath it? Always has been....

[@0xbyt3z]

any advice for pipelines for a nx monorepo

[@Jabberwockybird]

Avoid monolithrepos

[@morwar_]

JIT is GIT nemesis.

[@aCoreyJ]

But it's called a JIT compiler

[@Marque734]

Only listened with one ear but it sounds like plenty of complexity added just because js is not the right tool for the job.

[@themilkman3118]

One weird trick mentioned

[@0w784g]

Company could have sacked the finops team and saved even more money tbh.

[@qwfp]

Elm mentioned

[@yassirzerhouni8296]

Go Brrrrrrrrrrrr

[@nicgeorge6126]

Using FaaS for this use case is the skill issue.

[@bluegamer4210]

yey

[@Tobarja]

There are worse things to have on the brain.

[@mjsagxy]

If you don't know how you are going to test it, you don't know how to write it

[@luizgrocco]

Over and over and over and over and over again...

[@vincentv8991]

FinOps 🤣 you mean accounting?!

[@MayyasNakhli]

Under 1 second gang 😂

[@FromRootsToRadicals_INTP]

Aint nothing wrong with tities.

[@Reversed82]

i think the argument against javascript is probably a moot point here, i don't use checkly but i'm gonna assume the reason they use javascript is simply that people will write their own custom functions that will run directly on the container, so a) it has to be interpreted unless you want to compile customer's custom functions and then run them, which comes with its own overhead, and b) javascript is probably a decent tool for the job if you want to write checks that validate the integrity of some downstream service you are operating...

[@SRG-Learn-Code]

meh

[@housemana]

see they're doing actual programming unlike u lmao

[@krishnabharadwaj4715]

One of your best videos. Also practical. I don't like majority of the videos because they always talk about go/zig/rust.

[@0dsteel]

aaand it's a package.json issue

[@theairaccumulator7144]

Me when installing 12gb of dependencies isn't instant:

[@DenDodde]

@@theairaccumulator7144 You forgot the: ¯\_(ツ)_/¯