Bringing the Unix Philosophy to Big Data by Bryan Cantrill: FutureStack13

Nearly forty years ago, the Unix philosophy fundamentally changed the way we think of computing systems. Today, big data looks much like the operating systems landscape in the pre-Unix 1960s: complicated frameworks surrounding by a priesthood that must manage and protect a fragile system.
We will describe and demonstrate Manta, a new object store featuring in situ compute that brings the Unix philosophy to big data, allowing tools like grep, awk and sed to be used in map-reduce fashion on arbitrary amounts of data.
Be sure to subscribe and follow New Relic at:
/ newrelic
/ newrelic
/ newrelicinc

[@jmpalacios]

10 years later, still loving how comfortable Bryan feels being so inappropriate 🤣all the while delivering such amazing technological solutions!

[@tractatusviii7465]

I love Bryan Cantrill dearly

[@graphicsRat]

Brilliant speaker. Very compelling. Great talk!

[@willmcpherson2]

This guy is a wizard

[@idokwatcher2062]

So what makes your tr (or even better, grep) "parallel and distributed" is not that it somehow runs N greps per location to utilize multiple cores and whatnot, but just the fact there are as many individual grep processes running as there are objects to be parsed (307 in this case)? But what if your log file _is_ that mythical 700GB haproxy access log that is slow to copy back to you? Are such large files chunked into numerous "objects" residing on on many different ZFS servers when they are initially being sent to the object store?

[@adarqui]

great talk, loved it.

[@nickr753]

I think what I’m confused about here is where the compute is running. Like, are you not spinning up a jail on the same OS that is hosting your ZFS volume? Isn’t your storage server going to have different hardware characteristics that lend itself better to storage than to compute?

[@hectorjazz7]

yes.. ! bryan is a brilliant speaker

[@WilliamPayneLondon]

Great talk -- and funny too

[@edgeeffect]

"Goes 19th Century on my ass" :D :D :D

[@ChristopherOkhravi]

Damn hilarious :)