Building the Software 2 0 Stack (Andrej Karpathy)

A lot of our code is in the process of being transitioned from Software 1.0 (code written by humans) to Software 2.0 (code written by an optimization, commonly in the form of neural network training). In the new paradigm, much of the attention of a developer shifts from designing an explicit algorithm to curating large, varied, and clean datasets, which indirectly influence the code. I will provide a number of examples of this ongoing transition, cover the advantages and challenges of the new stack, and outline multiple opportunities for new tooling.
To learn more: databricks.com/product/gettin...
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: databricks.com/product/unifie...
Connect with us:
Website: databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / databricks
Instagram: / databricksinc Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. databricks.com/databricks-nam...

[@ExcessumGaming]

This should be seen by a lot more people!

[@godspeed133]

How had this got under 20,000 views and under like 10 comments?? This guy is literally one of the leaders in the industry, he's right at the bleeding edge. That's utterly diabolical

[@paulcassidy4559]

Compare with Ben Goertzel's discussions with Charles Hoskinson where they're literally openly sharing their ideas and thoughts around the bleeding edge in decentralized tech and decentralized AI applications with the entire world. Also getting low 5 digits view counts. Oh well. Just means people like you and I are ahead of the curve!

[@wbrandler]

Only a small number can be at the bleeding edge by definition!

[@TA-ob6kz]

Tools for Datasets writing software.

[@ihateyourusernames]

​@@paulcassidy4559 agreed :) most of us are here due to a shared sense of foresight. good luck to all of you on your projects :)

[@compact6264]

I did labeling job in the past and this guy described it perfectly, I feel better now for sucking at labeling :)

[@tipoomaster]

"Elon was like, vision can see raindrops...And now it's my problem" I feel like that's a fair snapshot at life in the FSD team lol

[@DogaOzgon]

This was very insightful, definitely more people needs to see this!

[@chrissanchez1891]

Ahead of his time. Very thought provoking.

[@Tony-cj6jy]

6:45 This is a nice proof of concept to me and explains why they could make progress much faster. First is an example of a hybrid heuristics approach (used by Waymo and others) and a pure deep learning approach (used by Tesla). The second one is superior in recognizing parked cars but requires massive amounts of real-world data. Only Tesla has that real-world data, because it's system is installed in about half a million cars by now. All the other "experts" thaught it was too dangerous to test self-driving software in cars being driven by consumers. They also thaught it was necessary to have Lidar, still too expensive to apply on a massive scale.

[@SyeamTechDemon]

Actually, Tesla does use heuristics for certain things, not everything is done through deep and machine learning approaches. Elon Musk says this himself during Autonomy Day, that sometimes it's quite stupid to have a neural network for simple things, you might as well just use heuristics.

[@rafaellino7168]

@@SyeamTechDemon makes you wonder about this NN for running the wipers then... maybe he thought it would help the NN see through rain if it could recognize it

[@MuscleTeamOfficial]

Karpathy, you will always be a legend in my book.

[@eduardorivara6129]

Great talk!

[@GoogleUser-ee8ro]

This whole idea of software 2.0 where we human define the scope/region/boundary of the problem and let the program/machine/AI find optimization(s) for the problem reminds one of how we approach PDE problems.

[@Noiseofdrums]

Andrej Legend Karpathy

[@Level6]

Good!!

[@liangcity]

very good video

[@IvanGarcia-cx5jm]

There is a book named "Working Effectively with Legacy Code" for procedural/algorithmic/classical (software 1.0) code. It would be nice to see a book like this for statistical based/deep learning networks code (software 2.0). I guess since all of this is relatively new there is no deep learning legacy code yet (although companies like Google might have a bunch of it already). But it would be nice to see how maintenance and refactoring would be done in not only in legacy code for NN, but also legacy network parameters that have been calibrated with potentially years of training (if that exists). I am not sure if this really matters for software 2.0.

[@wimveninga1714]

Wouldn't the "maintenance part be" on the data, the quality of the tagging and the process of continuously improving the model based on new data?

[@IvanGarcia-cx5jm]

@@wimveninga1714 Makes sense. The maintenance is on the data to train the model. But it seems to me that the weights of the network are unmaintainable. The only things a DL engineer modifies are the model and training data, hence, that is the only thing to maintain. Maintaining weights are like maintaining binaries, it does not make sense.

[@jimmyt_1988]

So well said. Visionary. Let's do this guys!

[@Vale46NL]

who is this guy man

[@quosswimblik4489]

Could AI recode the net for more slower rural users. Could a company have a lite site maker. If I wanted to get rich I would make an eternal search engine that was always learning about what it less understood to autogenerate labeled datasets.

[@scottdavidcraig]

🎯

[@colinmaharaj]

I wish I could work there. But I dont even have a 1st degree

[@Trashbag-Sounds]

Why don’t you just use the user wipe input to train? If there is ketchup on the windscreen, the driver will use the wiper stick to wipe. To you can detect this as an “automated system failure” and retrain on this. End to end. Isn’t the pretty straight forward? Even for gals positives. If it gets to excited in a tunnel, the driver should be able to turn it down or off. So just use this as an intervention. You do this with autopilot. Why not with the wipers

[@SuperChooser123]

Well they probably have it a little harder than just throwing a bunch of pictures of a tunnel rear view to the trainer

[@Splish_Splash]

it doesn't solve the problem with tunnels

[@Trashbag-Sounds]

@@Splish_Splash of cause. The user disables the wiper and this data is collected. So this way tunnel data gets send to Tesla in order to train on these .. automatically

[@Splish_Splash]

@@Trashbag-Sounds but you will need huge amount of time to collect this data with tunnels and wipers, and they don't push new release of autopilot if it doesn't properly work

[@Trashbag-Sounds]

@@Splish_Splash well I assumed they already had a version on the road that wasn’t working properly

[@TA-ob6kz]

Tools for Datasets writing software.

[@openroomxyz]

Are people who annotate your data paid decently?

[@rafaellino7168]

hahahaha

[@frankhandroid1252]

hmm. sounds like "we have no idea why it works or when it will not, but we sell it anyway"... Even a wiper can become safety relevant if it starts wiping like mad at the wrong moment, distracting the driver (as long as there is one).

[@Splish_Splash]

there's a thing naming "testing" and "shadow mode"

[@kythrathesuntamer9715]

I want to understand what he means by programming 2.0 so i'm here.

[@frankhandroid1252]

Also shouldn't dub it "2.0" it "1.0" is not replaced...

[@blackthirt33n]

Andrej karpathy

[@andzrit]

Whole issue is: the presented notion of Software 2.0 collapses on it self. Suddenly the data is the entire program? No. The data is still the data. Software 2.0 is essentially Data 2.0 Software 2.0 would incorporate Data 2.0 and Networks 2.0 and so on Reactive, layered programming style with a functional category approach probably makes most sense for the Programming 2.0 of Software 2.0

[@Splish_Splash]

functional programming bruh

[@Vale46NL]

A problem does not exist

[@Vale46NL]

To solve aproblem you must see no problem

[@Vale46NL]

A problem does not exist

[@thomasharrison2741]

08:42 - You know this is literally the secret to how God runs the Universe, right? lots of data, few rules. the Universe figures it out.

[@maqboolfida786]

Why not use AI to label data? Won't that be more accurate?

[@rafaellino7168]

you train the AI to label the data by having humans do it first. AIs have no idea of what they're looking at - they're (mostly) just smarter data processing engines using Big Data human input

[@Vale46NL]

wrong!

[@Vale46NL]

program space consisting problems is a waste of space

[@Vale46NL]

this is all wasted time thinking about

[@Vale46NL]

he doesnt say anything actually. just like stupid ai

[@444haluk]

Intelligence isn't about struggling to differentiate between the droplets a few cm away and the sun a few light minutes away. The end doesn't justify the means. Your company's entropy reduction methods suck. Big time.

[@444haluk]

This dude be like: do not try to explain it, it works... most of the time. What a naive problem solver.