What is Big Data? - Computerphile

[@neuron1618]

big data is anything that is too large to be opened in excel

[@senkottuvelan]

True.😂

[@Rakkoonn]

You say that, but many companies who say they use 'big data' really mean a huge spreadsheet.

[@ubummer]

big data is anything too large to fit in pandas

[@minihjalte]

For thats there is microsoft access. Its just large excel :^)

[@michaelsommers2356]

_"big data is anything that is too large to be opened in excel"_ So big data is any table with more than 100 rows?

[@BaronSamedi1959]

According to management all big data can be reduced to one nice coloured 3D-pie chart!

[@Walleggwp]

And if you don't have a nice upsloping line graph, well... I'm sorry but I'd like to speak to you in my office when are you finished.

[@napillnik]

@@Walleggwp hockeystick!

[@napillnik]

mmm, pie... I keep suggesting it but my team starts ignoring me after that.

[@Manabender]

More V's of data! *Volatility*: How likely is it that this data is received intact? How often do the bits get flipped? *Velociraptors*: How much would this data scare xkcd? *Vaingloriousness*: How hard is the creator of this data trying to shove it in your face despite repeated attempts to get them to shut up? *Vanity*: How likely would the data be to win a beauty pageant? *Vampiricism*: When mirrored, does this data delete itself? *Vaccination*: Has the data been protected from viruses? *Vuvuzela*: Honestly, this one should describe itself.

[@jackik1410]

this is glorious! nearly died here XD

[@triton62674]

Superb haha

[@letMeSayThatInIrish]

Vastness: Does 'huge volume' not even begin to describe the sheer size of the data? Verse: Is the data in verse form? Viscosity: Does the data flow effortlessly, or does it lump up like blood clots? Vikings: Does the data contain false information about vikings, such as them wearing horned helmets? Vendetta: Is the data vengeful? Viciously vindictive? Vincent van Gogh: Is it art? Vortex: Does the data rotate in ever more violent circular motions around the data center? Vulgarity: Must the data be censored for people in the US? Violas: Would a symphony orchestra make fun of the data?

[@jackik1410]

@@letMeSayThatInIrish Holy data, this is even more ridiculus. The beauty is that each of the makes so much sense by itself and represents an actual (kinda) valid query!

[@SniperSpy10]

Virginity: is it new and pure Violence Level: how likely is it to destroy other data Vocal: how easy is it to be heard Viagraity: can it give the reader a hard on

[@macchicken98]

Hands down best explanation of big data I have seen. I‘m coming from a business degree where we often learn about the 5Vs but don’t really touch on what infrastructure is actually used or needed for using/handling big data. Now I definitely have a better perspective on this!

[@koz857]

I picture a computer scientist somewhere thinking "Hmm gravity of the data is an important aspect that should define big data." and his friends are like "It doesn't start with a 'V' it won't work"

[@galgrunfeld9954]

Value of importance - how important the data is Based on the value you can manage its position in a data pipeline - e.g what dataset you process first, how much computation power going into processing it, what data is sent to nodes in a network first, etc.

[@Epic-so3ek]

I'm pretty sure them using all v's is to appeal to people who don't have a computer science background (aka managers and execs), or maybe people taking a first course in data science. I don't know that for sure but just the fact they used "velocity" instead of throughput makes me think that. If it was for people with a cs/IT background, that would just confuse them.

[@allluckyseven]

TIL a little bit about Big Data, but also learned that in England a truck is called a lorry.

[@randallanderson4999]

And a highway is called a motorway.

[@a.yashwanth]

In India too.

[@Jamie-st6of]

Ande Yashwanth well yeah, cause england invaded india

[@lsmeteor4652]

And in the us, you park on driveways and drive on parkways

[@NoseyNick]

That's nothing, they come in different colours (with a u) too! Try saying "red lorry yellow lorry red lorry yellow lorry red lorry yellow lorry" really fast.

[@senkottuvelan]

8:32 Sean Ridley is an awesome editor. Used the word Process to add Pre Process in the video.💯🔥

[@MILCHMONSTER3D]

my modded skyrim is big data too much for one computer to handle

[@hattrickster33]

I know what you mean. I literally have to run the game at my local rendering farm to get anything over 10 fps.

[@manualvarado2212]

@@hattrickster33 At least you have a local rendering farm.

[@edgekane958]

Every Computerphile video deserves a like. Change my mind.

[@jasetran2186]

nope. you're right sir!

[@code-dredd]

"Big Data" is the confusion that follows after marketing people end up describing technical stuff.

[@WilliamAncich]

Could not agree more.

[@cmonkey63]

Did you know? The term "Machine Learning" was an invention of the marketing team at IBM in 1959. Machines don't learn, silly. Well, neither do people, much of the time.

[@MrCmon113]

@@cmonkey63 Machine learning describes precisely what it's about. Really, I cannot think of any better term for it. Computer aided reverse deduction? Knowledge discovery in databases? Automated stochastical analysis? Practical function fitting? Those are all obscurantist, *learning* is what it's about. And who learns? A machine.

[@alkis2407]

@@MrCmon113 Statistical model estimation/fitting would be more accurate IMO. Optimization has been around for ages, why call it learning all of a sudden? (hint: money)

[@napillnik]

@@alkis2407 algorithms learn. They adapt without code being rewritten, and produce outcomes that haven't been preprogrammed, and get better with experience. That's learning.

[@Shadow81989]

Great to see more of Rebecca! This one was much better presented, seems like she's getting some practice (and confidence). :-)

[@AndyH2O]

...and is being patronised slightly less.

[@sumitrana8114]

Let's take a moment and say that computerphile never disappoints.

[@fcs_96]

This channel is super informative. I'm super pleased that I was able to stumble upon it. Broadens my knowledge of Computer Science.

[@RichardT2112]

It’s not the size of your data that matters, rather how well you process it ...

[@MrCmon113]

No, it's both. We knew about lots of the best machine learning algorithms more than thirty years ago, but we didn't have the datasets to train them sufficiently. Deep neural networks are comparatively simple, but they perform miracles if you throw tons and tons of data at them.

[@RichardT2112]

Taxtro I see humour isn’t lost on you ... thanks for playing along!

[@Monk-E]

@@MrCmon113 wow you're cool

[@Bnelen]

She does a good job of covering many of the important basic concepts.

[@MrFloris]

Thank you for making these and sharing these lovely videos. They're a fantastic resource.

[@stefanjooste3598]

Love the use of old dot matrix printer paper to try and explain the basics of big data.

[@ShankarSivarajan]

A quote I heard last week about big data: "We are drowning in data but starved for information." (Paraphrasing John Naisbitt, 1982).

[@MrCmon113]

Information is just the complexity of the data. What you are looking for is knowledge.

[@ecelon]

Big data for me is when any text editor I try crashes while opening it...

[@DantalionNl]

Can we also get videos on big data using none Spark based technologies?

[@noredine]

It's the opposite of ˢᵐᵃˡˡ data

[@adityasanthosh702]

rofl

[@PaulaJBean]

How do you make KZbin render small text?

[@noredine]

@@PaulaJBean Google tiny text

[@PaulaJBean]

@@noredine ᵀʰᵃⁿᵏˢ ᶠᵒʳ ᵗʰᵉ ᵗᶦᵖᵎ

[@Bordsteinpflaster]

I started to research to that topic today and was even on this yt channel to search for stuff ... and tadaaah I see this upload in my subbox, perfect timing :)

[@DavidLindes]

Good stuff. While I knew each of the concepts, I'd not heard of the "5 Vs" (let alone the 10/whatever)... cool! And wait, is this map/reduce video out already? Must find it. I've been wanting a refresher, because I haven't used it in a while, but it could be useful for me soon.

[@fruitfcker5351]

01:28 I haven't seen that wide of a continuous paper in decades

[@uristmcdani]

Thanks a lot for this explanation, very clear!

[@quratulain8396]

Productive video

[@raffriff42]

KZbin views and likes are tracked by traditional databases. KZbin recommendation algorithms use "big data" (although they use views and likes as raw input) "Big Data" systems are mainly interested in the _patterns_ in the data (data = whatever information is fed into the system), and the integrity, or confidence in, the individual atom of data is not very important. OTOH, in traditional databases (bookkeeping, inventory, payroll) the integrity of each atom of data is (with some exceptions) very important indeed.

[@vedi0boy]

Looking forward to the next video, thanks!

[@RAZREXE]

Big data is the study material folder in the d drive

[@moni7235]

Thank you Rebecca!

[@johndripper]

i can listen to u all day :)

[@kevind814]

Big Data: The lifeblood of Big Brother

[@robertboran6234]

Long time ago i was thinking that we can in theory use Big Data to create new electrical energy that can feed other machines or even the Big Data system itself. When we have huge amount of data, some of it is relevant information (this is used for processing) a second type of data is a second relevant data (this is used to train the Big Data system to improve itself) and the last type is total garbage data (this is still data that has 0 and 1). Now we know that when digital information is deleted from the machine the actual bits of information are not lost but transformed via thermodynamic effects into heat (this heat is raising the temperature of the machine) so when digital data is deleted the machine will heat up a little bit. Now we channel all the heat from all the machines and instead of disposing it we reuse it to produce electricity. So we recycle the "heat" from the machine.

[@TheSam1902]

But you forgot something, it's not the heat that is valuable, it's the heat **differential** . Some datacentres in northern countries uses the temperature difference between the inside of the server room and the outside air to power Sterling engines and produce electricity, but it's still not very efficient. Also iirc the swedish military won a wargame against the US because their submarine were (partially) powered by these Sterling engines making them stealthy than nuclear/diesel powered submarines.

[@robertboran6234]

@@TheSam1902 I agree with the inefficiency. Another way to improved this is by increasing the information density. But i still believe that this will be possible if the system is large enough. I am thinking about interplanetary internet where you need to process all the data of an entire planet. Also we know that information at a quantum level is stored in the surface not in volume. so i am thinking of using black holes as memory.

[@lmaoukiddin680]

3 inches is pretty big right?

[@sooskca]

How many Apache projects are there?

[@AndyVanee]

At the moment... exactly 367

[@TheSam1902]

As much as the number of feathers on a peacock.

[@gorgolyt]

There's only three Vs, the last two were clearly added on because somebody wanted five "Vs" but they really have nothing to do with whether something is big data.

[@TagetesAlkesta]

Big Data is a great band 👍

[@sabuein]

Thank you.

[@dancingCamels]

Step 1: Rotate/hone the rocket Step 2: Light Step 3: ... Step 4: Profit!

[@NoseyNick]

I think it's "rotate / move the rocket" but I hope we learn more about Rebecca's Rockets in a future computerphile video!

[@dancingCamels]

@@NoseyNick oh yes, on looking again you're right. Hopefully we will find out what it's about!

[@nathangek]

That's data but, like, really big.

[@dirkdigglerswonderlandempo5170]

How times have changed in my day it was the 4F's now its the 5V's

[@hillwin10]

Does size really matter? It is how the data is used. edit: or "data are"

[@michaelsommers2356]

It depends on whether you are referring to the data individually or collectively.

[@thomaspearson8782]

@@michaelsommers2356 wouldn't you use datum if it was singular, and data otherwise, using "is" for both?

[@michaelsommers2356]

@@thomaspearson8782 Sure, but I was mostly joking.

[@MrCmon113]

Ok so you have to tell me what distribution produced the following input-output pair: A -> 0 Do you think your chances of guessing the right function improve if I give you more examples? If not, why do you think learning is even possible?

[@farqueueman]

"how big is big" giggles

[@jvne_]

"How big is big data?" Me: big

[@strydomobile]

Lorries are awesome.

[@rock3tcatU233]

It's not the size of the data that matters, but how you use it.

[@MrCmon113]

The size of the data matters a lot. Some things you can only learn from incredibly huge sets of data.

[@rednull8315]

640 kB

[@satyris410]

That's a big excel file

[@maulanaibnusabil5280]

Can someone explain me the difference between Big Data, ETL (Datawarehouse), and Data Engineer. I'm really confused

[@Treviath]

Would it be possible for you to do a video on the piece of art that is called Wireguard?

[@gabetower]

I won't be content until you have more V's than the speech from V for Vendetta. Voila!

[@MoonMarshmallow]

Rebecca is so cute!! ❤

[@polygondwanaland8390]

@MichaelKingsfordGray What's your address and credit card number? Wouldn't want to be anonymous and cowardly, big man.

[@inzanozulu]

Use your inside voice. It's not a problem to find somebody attractive, but did that really need to be in a comment on this video?

[@olik136]

I think data has to be at least this >| |< big... maybe even this >| |< big...

[@edge4694]

I hate how the sound of the pen lags behind the actual pen

[@rendogsbiggestfan]

I didn't realize but now I can't not realize, you monster

[@StrangeIndeed]

I've realized that 5 V makes for a very nice mnemonic. V is 5 in roman numerals, so you can pretty easily remember that there are 5 Vs. It's probably just an accident, but makes it things a little easier to remember c:

[@laser-sj]

Mind tickled :p

[@LathosZan]

Always like for gals in tech!

[@axcelleria]

Well she's sweet

[@snake1625b]

Generally, more than 10 terabytes is big Data usually

[@Alex1891]

When I was a kid, I used to say things priced at $30 or greater were expensive, regardless of context. ;)

[@snake1625b]

@@Alex1891 most things in life are subjective and don't have a definitive answer. But it's definitely possible to give a generalized average answer. In this case you can say the AVERAGE server can only process less than 1 terabyte of typical data and thus you'll need multiple computers to process the data. The most unhelpful and pedantic answers you can give is something annoying like " it depends. It's subjective. It varies from problem to problem".

[@BlackHermit]

big data is anything that is too large to be opened in excel

[@realityveil6151]

Hey, she's back! The cute nerdy chick!

[@mannycalavera121]

I like to pretend i'm smart enough to understand what's going on in this video :)

[@kennb33]

Isilon

[@JanB1605]

How I love me some pretty, intelligent women in STEM. Great Video, was always wondering what big data really is.

[@xakkep9000]

coool

[@grainfrizz]

Vig data

[@Ubeogesh]

so where's that map reduce video?

[@misterhat5823]

She's adorable.

[@willhendrix86]

In before your entire life and your rights are represented in a 5 star rating system; And yes I have seen that black mirror episode ( ' ', )

[@gqh007]

In before killer robot bees

[@BrokebackBob]

Data storage is now totally separate physically from the computers that access it. The idea of defining big data as the max that a single computer can process is laughable.

[@szebohalasz7793]

I dont thik so, given the fact that you mostly need the "computer" to process the data. Also its just metaphorical not absolute definition, as the BigData itself.

[@senkottuvelan]

KZbin IS BIG.

[@mtranchi]

gah, too cute

[@SephirothDL]

Splunk

[@MarcelRobitaille]

A buzzword?

[@rogerc7960]

NSA/GCHQ giggles at your ignorance

[@alittlebyte]

00:01 "How big is big?" LOL

[@mochrDK]

Do you pronounce it Data or Data?

[@sensibleb]

Star Trek TNG changed the way I pronounce 'data'.

[@iLiokardo]

big data is management and measurement of private data

[@nosuchthing8]

Lorries are trucks in pondese

[@TheGreenTaco999]

Big data energy

[@pajeetsingh]

I want to **** *** *****

[@catklyst]

BIG DATA ENERGY

[@vjndr32]

Can you please stop writing on paper with markers or atleast suppress the sound of it? The sound gives me goosebumps while watching videos. Doesn't it happen to anyone else?

[@mohammedmohammed519]

Vijender Singh No.

[@maggieselbstschopfer1956]

0:02 How Big is Big!!! My ex BFs mmmm..

[@AnimilesYT]

Big Data? You mean that folder on my computer called "internet explorer"? xD

[@billykotsos4642]

Yeah BABY

[@BrandonJohnson2k7]

the lack of technical words being used to describe computers and the database hurts. relational database, datacenter, server

[@RatnaMaharaj]

The nerds really need to learn to communicate well, otherwise the likes of Zuckerberg will keep winning.

[@jonasgrnbek7113]

wtf haha

[@MrCmon113]

What's wrong with Zuckerberg? What nerds are you talking about anyways?

[@AudioPervert1]

No one seems to Talk about the huge amounts of energy and electricity needed to churn all such big data (all fossil fuel based) ** the big data can stop the sixth mass extinction ?? Carry on technology based omnicide.

[@jmonsted]

Our datacenters were pretty much entirely wind powered. Many of the big datacenters try to make deals to buy 100% green power.

[@polygondwanaland8390]

Where do you get the idea that datacenters are "all fossil fuel based" when there's entire states, provinces and countries that don't run any fossil fuel on their grid?

[@hoola_amigos]

This is a relatively poor computerphile video.. all theory and bookish explanations.. I didn't take home satisfactory amount of knowledge.. Wouldn't be able to explain anything (except for 5 Vs) :/ Maybe it is the topic itself which makes it very difficult to practically explain it to someone who isn't familiar with all those terms..

[@mini_frank]

when he said how big is big, my first thot was the unit inches...

[@mountedczarina9205]

She’s very cute.

[@bbuggediffy]

No that's not the definition. It's the CAP theorem that defines it.

[@Ashkenya]

Sorry but there is no such thing as Big Data. There might be big database, or big chunks of data, large amount of data, heaps of data. But data in computer term is just a collection of what we humans classify/distinguish/label as "1s & 0s"

[@ivahardy4885]

but is it pronouced data? or data?

[@roshsiddiqui8745]

you are so cute and adorable, i am in love cuz along with all of that u are into tech and have a techy brain

[@iiiiii-w8h]

That's a funny looking kitchen

[@omri9325]

The definition is really lame

[@joonasfi]

You can make any data big data by exporting it in XML

[@Lodinn]

It's hoooman-readable format though amirite? :>

[@satyris410]

More than 16,384 columns = Big Data.

[@derpimusmaximus8815]

"This data is small, but the data over there is far away."

[@recklessroges]

Thanks Ted.

[@bencrossley647]

Best / most unexpected comment I’ve ever laughed at. I can see him looking so confused.

[@willynebula6193]

Candy crush is big data for my Amiga 500😉