Code vs Data (Metaprogramming) - Computerphile

Why are code and data so separate? Robert Smith of Rigetti Quantum Computing explains how he uses Lisp code to generate Lisp data which he can use as Lisp code...
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottsco...
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com

[@user-zz6fk8bc8u]

@2:52: *Reversed reverse Polish Notation* or as sane people would call it... polish notation ;-)

[@SouravTechLabs]

@science no it's at 2:50

[@user-zz6fk8bc8u]

@Sourav Goswami - Thanks. Fixed it.

[@SteveHodge]

Or as people who prefer descriptive rather than arbitrary names for things, "prefix notation".

[@alanvitullo]

Dude, please start a programmer's channel with a LOT more code. These videos are a tease.

[@debjyotibiswas3793]

Agreed. Show me some code.

[@greenlight2k]

If you want more code, check out "The Coding Train" channel ;-) Its... Javascript... but there you go...

[@TheCandyDragon]

aah, that channel is so fun

[@debjyotibiswas3793]

Yes Daniel Shiffman is very cool. But hate JS are there any other channels which are not JS.

[@debjyotibiswas3793]

Yes and JS is very easy to understand. The problem arises when JS is used interchangeably with node and starts doing weird things.

[@ninjalacoon]

Linear algebra has this same kind of duality between data and function. A matrix is just a collection of data in most cases but when you multiply a matrix by some other matrix or a vector the matrix acts as a transform or a "function/code" on the data that passes through it. A change of basis is a transform matrix that acts as a function while at the same time could be further transformed by rotations and scaling matrices and act again as just data.

[@Luredreier]

Think of meta programming as inventing something new like multiplication or even pluss in math prior to those things being invented. Or any other new concept. For instance, there was a time when no one had invented zero. Math at the time had to be done with numbers with a value since there was no zero. In a programming language with metaprogramming you can invent such a new concept as zero, define it and use it instead of defining it every single time you need to use the concept.

[@Evan490BC]

Well, almost. A matrix is a *representation* of an operator relative to the given basis.

[@pablo_brianese]

Love this comment

[@OverThinkKS]

Please learn about geometry algebra

[@ToveriJuri]

@@OverThinkKS Linear algebra taught me to like math

[@tsunaminoai]

"So could you give us an example?" "Oh sure, definitely. So lets say you wanted to draw an owl. First you just draw some circles. Right? Then you draw the rest of the owl, which I wont do here because its obvious."

[@qwertyman1511]

as i understand it: first, a function is a set of instructions. second, we can create data during runtime. third, we can interpret data as instructions. fourth, we can run interpreted data as a function. you created code at run time. the program wrote parts of itself.

[@cj37373]

Andrew McAuliffe you can base code on input from the user if you desire so

[@ElagabalusRex]

The way he described it, metaprogramming is the same as creating a library, which doesn't sound right.

[@quadricode]

ElagabalusRex you can create libraries of syntactic abstractions. That’s part of the point. :)

[@quadricode]

Kris2510 great, written out example!

[@sweetspotendurance]

I love listening to this guy talk, I don't know why

[@jaymalby]

Really nice broad pass over metaprogramming 👍

[@ArumesYT]

This duality between data and code is much more common than most people think. One very simple example: most of us view a .c file as code. But to the compiler, it's data.

[@Roxor128]

Exactly. The very existence of compilers and interpreters relies on that duality.

[@davidwuhrer6704]

But can a C program evaluate, change, compile, and assemble it's own data at runtime?

[@-notakil]

I want more from Robert Smith. This guy is good.

[@emilemil1]

To me that for-loop example seems very similar to defining a function that takes other functions as its arguments, though perhaps with a nicer syntax. Would love some more in-depth examples.

[@marios1861]

the for loop takes a variable name, the starting value, the ending value, the increment and the body of code to be executed with each iteration. Can you make a function that you give as input a body of code? Can you bend the syntax of the language you use in any way you see fit?

[@albertosimeoni7215]

Is the standard C++ loop... The right example in C++ for metaprogramming are Virtual classes / methods

[@KnownNever]

Am I correct to say that Lisp as he describes it is better off for math problems while C is better off for dealing with memory... cause that makes scenes why you would rather use Lisp over C for something like Quantum problems as stated in last video

[@vyrsh0]

All abstract problems, while C is better for memory as you said

[@scottfranco1962]

"reverse RPN" is not Reverse Reverse PN, it is just PN. Polish notation is like + 1 2 or add 1 and 2, RPN is 1 2 +. Polish notation was specifically prefix notation, which was considered more natural than 1+2 because you can write +1 * 2 3 to mean 1+(2*3), and indeed, this is a common notation used in compiler intermediate codes. However, RPN is considered even more natural than this because 2 3* 1+ exactly expresses operands and the operators that use them in proper order. Thus get 2, get 3, multiply, then get 1, and then add to the result of 2*3. It also is stack machine friendly. If there never were such a thing as polish notation, I'd agree that reverse RPN makes sense, but polish notation was a thing before RPN.

[@Roxor128]

Oh, dear. The comment-formatting system has gotten tripped up by your example and parsed the asterisks for multiplication as instructions to make the text bold.

[@scottfranco1962]

Thanks! Fixed it.

[@Roxor128]

I really wish they'd just use a subset of HTML or pseudo-HTML for the formatting. These errors would be so much less likely.

[@brianbarefootburns3521]

Would be interesting to see a clarification of how this topic compares to compilers in general -- because, of course, compilers are programs that write programs. I was waiting for the comparison and distinction to be made in this video, but it never happened.

[@vicplichota]

Forth is another wonderful language that still stands the test of time. This vid mentions Postscript, which is very much a variant of Forth.

[@grivar]

Duality: Tom Scott or John Green?

[@jaymalby]

RE now I can’t unsee it 😂

[@AlterMacGyver]

Trick question, Griffin McElroy

[@taba1950]

Arguably better than both

[@mal2ksc]

Tom Green.

[@grivar]

Or is is John Scott?

[@Majorohminus]

I'd love to see someone debug a metaprogram that contains thousands of lines of business logic :)

[@albertosimeoni7215]

During my job in a company i find a procedure that was modified costantly in 4 years ( a T-SQL store procedure) that has around 3000 rows with some part of code that write code that was executed... It takes about 1 week to under stand it...and about 1 time over 2 month it need to be modified...The source of this madness is called SAP

[@Alsteraib985]

A lot better than the Java mess.

[@juancarlosgallegos3902]

I did not really understand this, he seems to be skipping some important details in his explanation?

[@maestbobo]

I mean, it's obvious what he's saying. I don't understand what you're finding difficult about it.

[@ZeedijkMike]

I'm getting all warm and fuzzy when I see the _LISP_ syntax. Brings back good memories from when I used it with AutoCAD.

[@LuizBGomide]

How different is that from ANY other interpreter? Or writing instructions directly to memory updating code on fly?

[@TakanashiYuuji]

It's not. Most languages have some kind of 'eval' feature.

[@heller166]

yeah i also don't understand what is so special about it.

[@AMcAFaves]

It is a more abstracted way to modify code. You are modifying the code at an Abstract Syntax Tree level. (And with abstraction can come more power.)

[@cmckain13]

TakanashiYuuji 0ⁿ

[@nitowa]

The special part is, that the program doesn't have to be recompiled by the programmer to introduce new features, formally this is called "at runtime". Simply speaking your program is conjuring itself while it's running, and doesn't come with all the necessary code (as defnined in the video) but instead it has the data necessary to create the code. A popular counterexample would be C, which requires you to compile it for the given architecture before executing it. C cannot add (for example) a function that was passed from the keyboard as C-code, despite being valid C code, and it cannot be executed at runtime. If you're looking for an "interpreted" language, you might find Java to be a suitable example (I know it's technically compiled, but it's also interpreted at the same time, so there's that). Although Java may modify the JVM runtime however it likes, changing the functionality of a method/function is prohibited, which severely limits its at-runtime capabilities (in practice it's possible but produces very patternized code and complex structures)

[@APSStevens]

I will remain indebted for sparking my interest in Lisp.

[@safirswe]

I want to look at the video teased at the end, "fighting obsolescence", but can't find it anywhere. Anyone got a link?

[@BeastinlosersHD]

How is the for loop example (and how it is an example of metaprogramming if a for loop didnt originally exist in the syntax) different from inline functions or #define

[@udxs6782]

Isn't RRPN just Polish Notation?

[@aDifferentJT]

2:53 normally called Forward Polish Notation

[@defense200x]

This is a rather confusing example of the usage of Metaprogramming

[@quadricode]

Unusual Activities you didn’t have for loops before, now you do. Is it confusing because everybody already has them? Maybe a better example would be: Imagine you wanted to specify logical constraints a la prolog. You can use metaprogramming to add such a facility to your program. But that means you’d have to explain what prolog is...

[@webgpu]

that's what you get when you assign that "confusing" guy to present this video..

[@defense200x]

I mean, I get it, actually I would to turn data into code for a project I'm currently working on but the language im working in (C++) won't let me :p

[@zacharymetz9975]

you could implement a scripting language then you could execute instruction that were created

[@greenlight2k]

Hmmm... can i think about this similar to Lambda-Expression in C# or Java? Which, like... IS program code at compile time and BECOMES data at runtime? Or, is that too far fetched? ^^

[@jakejakeboom]

A lambda expression is a functional concept so yes, that's somewhat similar.

[@kwanarchive]

The comparison breaks down when you try to pick apart the lambda-expression - which you can't do in C# or Java. You can create a lambda, but you can't pull it apart and pick bits and pieces out of it. With metaprogramming, you would be able to compose functions together, and then later on, pick them apart and, say, recompose the functions in a different way based on some conditions for optimization. C# and Java have no way to do this.

[@shanehebert396]

Currying using lambdas is an example of how to achieve some of what the video talks about.

[@Luredreier]

You can then get data that becomes code at runtime too...

[@RoGeorgeRoGeorge]

Where's the rest?

[@acharris]

Is it possible to get similar meta programming capabilities in languages other than lisp? IE are there other programming languages that can do similar meta programming?

[@rafeu2288]

Rust lang is another language with a somewhat powerful macro system. But it seems Homoiconicity is one of the qualities that make meta-programming easier to implement.

[@glowingone1774]

Nim, im not sure about "rust" but people have built whole DSL's with it in nim.

[@laughingvampire7555]

this idea that has been championed by lisp was how the original Assembly programmers did the code, they wrote Assembly to write Assembly because RAM was very expensive in the past. However when computers started to get cheaper, then people started to get lazy and try to achieve the dream of a programming language that looks like English.

[@CypressPunk34]

wouldnt reversed reverse polish notation just be polish notation?

[@davidwuhrer6704]

Yes.

[@AB-Prince]

Could you do a video on how to group logic gates to acomplish complex actions, such as multiplication, division, and converting binary to different two exponent bases

[@davidwuhrer6704]

I think they already did, using domino bricks.

[@rafeu2288]

I would recommend you check the video game "Turing Complete" on steam, it goes there and beyond, allowing you to learn how to implement a full 8-bit Harvard Style computer.

[@lawrencedoliveiro9104]

5:37 Assuming the increment of i lies at the bottom of that squiggle somewhere ... otherwise you’ve just written an endless loop.

[@umblapag]

Interesting to learn a bit about lisp

[@rkpetry]

*_...but tight code is used repeatedly-on scads of data; (Amer. scads, Brit. jillions)..._* *_...on the other hand, the similarity shows in loop parameters, i++, list descriptors,..._* *_...((in the control/addressing variables-but not, in the data itself/results per se))..._*

[@adamlanghans]

It doesn't seem like Robert Smith understands what he is saying. There wasn't much of an explanation here.

[@donotsellmydata2089]

To iterate is imperative and to recuse is functional?

[@BeCurieUs]

Hmmm still not getting it, I hope there are more of these.

[@CaptainJeoy]

It's just a method of abstracting useful functions as an evaluable data

[@inomo]

@Computerphile can we have a video about the Julia programming language?

[@habtamuassegahegn4283]

hey phile can you explain exactly about symbolic data from the book structure and interpration of computer program

[@daniellindner355]

is all of this related to code? like Metaprogramming and/or Software defined Networking? Where can i learn these things? Are they being taught in Computer Science?

[@noxabellus]

I would just like to suggest to anyone that is interested in this, check out terra lang :) It's basically a scriptable compiler and deserves more love

[@Eichro]

I'm assuming that in Lispland eval isn't evil?

[@surabax]

You don't use eval directly in Lispland, you define macros instead.

[@capoiosct]

nim-lang.org can do that and its procedural.

[@AMcAFaves]

capoiosct If that is the case, then nim wouldn't just be procedural, it would be multi-paradigm, because then you could just write macros to add OOP, functional or any other paradigm functionality to the language. :-)

[@ProjektRakete]

Is this the latest "Programmers are also human" video?

[@lawrencedoliveiro9104]

Von Neumann understood this perfectly well. The only ones who have forgotten tend to use languages like Java or C#. Oh, and PHP.

[@GreyDeathVaccine]

Don't be a hater. Use the best tool to solve problem at hand.

[@chpsilva]

​@@GreyDeathVaccine couldn't have said better

[@dickjohnson4447]

5:37 this is what you had to write to iterate ten times, yes but in assembly you could: times 10 instr instr (some instruction you want to do 10 times)

[@zacharymetz9975]

Whats the difference between lisp and a langue where i can run the eval() from javascript?

[@cobblebrick]

Aren't compilers in general an implementation of metaprogramming?

[@simplylinn]

That "assembly" for the for-loop is not actually that correct. The evaluation of the repeat condition is actually made before you enter the loop in a standard for-loop. What he wrote down is more akin to a do-while loop, which runs trough the loop once unconditionally, and then repeats if the condition evaluates to true. Actually, it's a bit of a hybrid, because he set the iteration-variable at the top, and is still using an iteration variable.

[@hikaruyoroi]

I really like this guy

[@tacchinotacchi]

I'm not sure where he is going with this. So, you're not happy with the tools a language give you such as for loops. So, do you want to be able to make your code flow in whatever fashion you want? Use labels on C. Yes, it will be considered spaghetti code, because it is. Also a c source file IS data, although stored less efficiently. So is Java code, and so is Java bytecode and assembly. A smart enough program could write programs in any of these languages. The difference is that some are readable, and some aren't. Judge for yourself which are.

[@Luredreier]

It's not about not being happy about for loopes. It's about being able to *implement* for loops on the fly in a language where it doesn't exist already. Or any other feature for that matter. If you can come up with a new consept you can implement it and use user input as a part of the implementation if desired. You can change the code itself based on user input. And sure, you can write a program that can write code for its own code. However what about writing a program that can write the code used to run itself during the runtime? Regarding the spagetti code argument. Yes, partially. But it doesn't have to be.

[@NikiHerl]

Please try to get the interviewees to explain their subjects more concretely (i.e. with code samples)

[@RealCaptainAwesome]

I had a guy demo lisp for me yesterday for the first time, now it is everywhere.

[@YourMJK]

Can't you also create assembler code in a C program and then execute it? Isn't that the same?

[@kwanarchive]

You can't take the generated code and make it into data again within the language. You'd have to disassemble the generated code, recreate the C from it (which is hard due to optimizations) and somehow make it recompilable. The other way is for your C program to create a miniature VM to interpret byte code your program creates, but that, again, takes you out of the language.

[@MasthaX]

Isn't it true most programming languages have an eval function where you could pass data (string, var, number, etc) to be evaluated and executed?

[@notnullnotvoid]

Only some scripting languages have this, and compiled languages generally can't. Eval also usually operates on program text, rather than directly on a generated AST.

[@piotrz3403]

reverse reverse polish notation = polish notation

[@bimbumbamdolievori]

A very powerful tool which I lost control over on day 2🤣🤣🤣now i kinda see where the elixir quote comes from...

[@Mike-qt4fr]

Love this man!

[@SteinCodes]

So Macros are also a type of MetaProgramming?? right!

[@NiDeCo]

Haskell doesn't have for-loops though :P It uses recursion, it being functional and all.

[@liln4444]

Shout out Julia

[@m.h.6470]

so its like the eval function in javascript and php which have been around for ages... and are generally considered as "dangerous", especially if you use it with user input.

[@malusmundus-9605]

Anytime you handle data from an unknown source it's dangerous... you have to predict what might come and handle it.

[@m.h.6470]

@@malusmundus-9605 and therein lies the hubris... unless you restrict the data, you can't possibly predict everything and handle it - especially with user input. A hacker can use these weak points to execute code without your knowledge and gain access to systems or data.

[@lawrencedoliveiro9104]

Code is just data to a lower-level processor. What is the lowest-level processor? Is it the laws of physics?

[@esobrev]

i feel bad for the computer scientists who worked on programs writing programs like this. because their way was the much more elegant, much more beautiful and pragmatic approach to programs that write programs. but instead we just went with this huge mess we call GPT-3 and language models. now nobody cares about this stuff even though it’s some of the coolest programming science out there.

[@BitcoinJake09]

I remember programming in QBasic with subs and things like be for learning java and functions :p

[@abdulrahmanalzeidi3811]

this is somewhat similar to templates in c++ no?

[@jaymalby]

guild man similar- C++ templates generate code for you, but in very rigid and structured ways and you write the templates in a special templating language instead of using C++ code to manipulate other C++ code directly as a data-structure.

[@punkkap]

>Is there a simple example of that?

[@esobrev]

he literally gave an example

[@Fiercesoulking]

So wrong what he shows as data is for all I know already code , data refers to the values of the datatypes(he refers to them as data)

[@diegonayalazo]

Thanks

[@NitinSethi22]

Beautiful

[@user255]

Any actually useful real world examples of these features?? To me it seems the core idea of Lisp is now obsolete.

[@soufta5872]

Actually, everything is data (even machine code is made of "data").

[@procactus9109]

Subroutines don't count ?

[@quadricode]

ProCactus you can’t make a for-loop subroutine for example. Subroutines are a great type of abstraction, but they don’t allow you to introduce new *syntax*.

[@procactus9109]

I can with an index and an if.

[@noyb154]

data is to code what information is to knowledge

[@damejelyas]

Write programs that can write programs he means create a language and use it

[@Wizball]

A bit retro to use dot matrix paper ;)

[@DavidBadilloMusic]

Isn't this metaprogramming sort of what High-Level programming languages are?

[@FinaISpartan]

How is this any different from C's #define?

[@gracicot42]

Final Spartan you cannot have instructions that write defines and you cannot then generate code that are using those defines. Macros are expanded before even compilation starts. Templates in C++ are a bit closer to what is shown in this video, since you can manipulate types and their structure as data and output new types. For example, I have a function that takes a lambda as parameter (a lambda is simultaneously a type and it's instance) and that function returns a new lambda with slightly different parameters.

[@HrHaakon]

#DEFINE gives you text substitutions. defmacro let you tell the compiler what you want done with something. You can tell it how JSON works and from then onwards, you can give your compiler JSON and it will give you an object back out.

[@marceloconceicao2587]

#define is VERY limited on what it can do. You cannot, for instance, change/create a #define based on user input because #defines are preprocessors (They are evaluated at compile time (even before that, actually)) And even regular programming is very hard to do with defines. Try to pass a macro as a parameter to another macro . Spoliers: You can't. The preprocessor definitions are evaluated only once

[@MidnightSt]

how is this different from #define ? ...completely. define is basically a shortcut/automation of string search and replace. what he's talking about in the video is basically programmaticaly building expression trees (code as data), which you can then execute by traversing them.

[@PetrSojnek]

The better example would be function like printf. You fill in parameters that will tell you what you do with other parameters. Essentially the "format string" is data that defines procedure to be applied on the rest of data.

[@dannygjk]

I think it's meaningless to say that before high level languages For... did not exist. The only way that makes sense is to say that literally it didn't exist. Which is also meaningless really because conceptually it doesn't matter which language you use to implement the idea it is conceptually the same thing.

[@samsharma8621]

Wheres mike pound

[@WarZzed300]

You can also directly obfuscate your code for exemple, turning it into base64 data and create "eval" function that decode and read it. So powerful !

[@chenzhuoyu1992]

Arbitrary code execution built into the language itself

[@quadricode]

Zhuoyu Chen everything is determined at compile time, unless you use EVAL yourself at runtime.

[@jamma246]

I don't feel like I understood the point from the video. It just seems like a really ad hoc way of doing functional programming.

[@p.kay_____]

Woah woah woah

[@desmondwilson3416]

x is not defined...

[@lawrencedoliveiro9104]

4:16 But watch out for the security implications of this, if the code you generate was based on input from a potentially malicious user. Unfortunately, I think this is the fatal flaw in the whole concept. This is why it is better to build your own interpreter on top of the base language -- as you could, for example, in Python -- and then generate code for this interpreter from the data. That way you get more control over what exactly the code can do.

[@stoneskull]

i don't think you even need to build an interpreter.. there's plenty of meta-programming ability in python..or even using text files that can be manipulated having code in them and can be imported in..

[@franzlyonheart4362]

0:12 what

[@UltimatePerfection]

Really, did we need this video? To a computer there's really no difference between data and code. That's why old consoles under certain conditions execute data as code or use code as data (usually with an awful result, see Pokemon's Glitch City for an example or Missingno, really)

[@quadricode]

QVear Very rarely is “programs as data” a paradigm exploited in most popular computer languages.

[@rodbhar6522]

Lisp looks like an overly complex version of Forth

[@Pax_Veritas]

yep, mmhmmm, mmhmmm, yep, yea, mmhmm, yes

[@JohnsonLobster]

Damn that's a sibilant voice...

[@joyalmathew2156]

Did this guy bump his forehead?

[@brotherfiretribe9566]

What happened to this buddy's forehead

[@nateshrager512]

LISP: Lots of Irritating Strings of Parentheses

[@hereb4theend]

Large Insect Sucking People

[@darthnihiluz5305]

evals have never lead to bad things

[@tusharkuntawar6170]

(No offence) non-chad tom scott here

[@frankfahrenheit9537]

So you can use this stuff to extend the language? Add "for loops" ? What a great example. Actually nobody needs "for" loops if you have an "if" and a "goto " and an integer variable. After compilation, a for loop as any other fancy construct is converted to "gotos" at machine level.

[@DusteDdekay]

Actually nobody needs if and variables if you have an ldx, a dex and bne, but then again, nobody needs mnemonics if they just cared to remember their opcodes like proper men :D

[@jellyboy00]

It sounds very dangerous in terms of computer security XD

[@jellyboy00]

Well, I had tried to use php object to generate javascript for AJAX and use the same object to do SQL query the AJAX request in php. I kind of understand what he is talking about.

[@quadricode]

It’s not at all dangerous. It’s not like using eval() in JavaScript.

[@bopstarsstuff970]

good video

[@un2mensch]

That reminds me, I'm actually seeing The Cure in July

[@judgeomega]

reflection