Language Models are Open Knowledge Graphs (Paper Explained)

[@TheGreatBlackBird]

It is slightly comforting that even "high profile" papers often have meh ideas hold together with duct tape.

[@josephharvey1762]

I love the way you freely critique papers!

[@bennettbullock9690]

I have to read the article, but the Chomskian (conventional linguistic) perspective is that grammar is an aide to semantic interpretation, and therefore the LMM's knowledge of grammar is going to be definition encode knowledge of at least simple facts. Which leads me to wonder why we even bother with extracting a relation from attention matrices in the first place. Why not just extract them from the sentence itself, since the sentence specifies this relation?

[@karnigilon]

As you say much of this seems like "grammatical" discovery. How does it compare to simply using the verbs as a relation? maybe using the verbs as the basis , and then using the attention values to filter out some of the tuples can increase the recall?

[@YannicKilcher]

good idea, I guess the goal here was to publish, not to get the best numbers :)

[@sarvagyagupta1744]

I think you should make a video on the differences between these attention techniques and Lambda layers. Similarities and differences can be quite confusing.

[@horizon9863]

I tried TabNine but the CPU usage is really high, every time it starts CPU got 50% usage (AMD 3900X)

[@shengyaozhuang3748]

Many nouns will be split into word pieces right? how to compute attention weights for those nouns?

[@YannicKilcher]

good question, idk

[@TheZork1995]

I read that there is no way in combining the subword parts in a meaningful way. So maybe they just use the Subword embedding to represent the word. So a big word has more chances. But it's just a guess.

[@robertoc.martineza.3719]

2 years later: GPT4 can make nice KG using ASCII art or pixel art with specific color palette.

[@andres_pq]

Loved the South Park reference 17:30

[@DinHerios]

Click-bait generates more clicks, which leads to more advertising money. Or, as in the case of academia, potentially more cites, simply because more people read the paper. It was only a question of time before academia gets infected with this concept. It certainly seems to work really well.

[@G12GilbertProduction]

In SEO sites optimalization attended to search engine indexes it's probably being more copy only this same tags at same time.

[@ChaoticNeutralMatt]

Yeah. This wasn't what I expected and I'm somewhat disappointed. At least it was fairly immediate that this wasn't what I expected based on the abstract.

[@first-thoughtgiver-of-will2456]

is regressing on human input knowledge distillation?

[@YannicKilcher]

good point

[@SirPlotsalot]

There's a case to argue it's actually an example of optimally-robust curriculum learning which is neat

[@PeterOtt]

wow, TabNine doesnt even start auto-charging you after the 3 months expire? what a good guy thing to do!

[@herp_derpingson]

I think while this might work great in Wikipedia like well written corpuses. It will fail miserably for spoken human speech, as it is very noisy. But I like it. Simple idea but it will take you a long way.

[@DistortedV12]

Can you go from updated knowledge graph to language model again? If so, that would be really cool

[@skm460]

Won't it be an easier task to just form sentences from the triplets?

[@raphaels2103]

There is a method, rome

[@drhilm]

Even your commercials are interesting.

[@eduarddronnik5155]

Dawn Song is a seriously great name. Like Klara Himmel.

[@kavitachoudhary1112]

superb video .please guide or suggest new area of NLP from where i start and go for research.,thank u in advance please guide

[@YannicKilcher]

rare languages are a hot topic

[@kikimajo6850]

1⃣️ Knowledge in Corpus: Dylan is a songwriter 2⃣️ "Knowledge" given by spaCy: Dylan(Noun), songwriter(Noun) 3⃣️ "Knowledge" in pre-trained models: The word "Dylan" somehow relates to "is", syntactically or semantically. AND The word "is" somehow relates to "songwriter". 1⃣️2⃣️3⃣️ -----MATCH----> (Dylan, is, songwriter) (Dylan, is, songwriter) -----MAP-----> KG(Bob_Dylan, occupation, songwriter) It seems that 3⃣️ is not 'knowledge' but 1⃣️ actually is.🤔🤔

[@editorsworld1521]

Can't agree more, kind of overstating, especially its title...

[@konghong3885]

I would argue the model in the commercial is more interesting then the results shown in the paper (good explanation tho) KG suffered with so many problems (graph incomplete, graph retrieval problems) making it a nightmare to use in production this result is just again demonstrate the power of transformers to perform in fewshot tasks, even if the task is knowledge compression (KG)

[@florianhonicke5448]

yeahhhh, another week - another paper!

[@maxwellclarke1862]

I don't get the comment about Rust :) ?

[@sillygoose2241]

The relation must be in between the head and the tail?? Yoda pleased is not

[@manojb8876]

Wait, is tab9 even legal. Isn't gpt not supposed to be released to companies and now bought by Microsoft?

[@BryanRink]

It's based on GPT2, which is released. You're thinking of GPT3.

[@DamianReloaded]

EDIT: Can't gpt3 do this already? Like giving it a text as prompt and get it to generate an xml containing the triples in the text you gave it? O_o

[@ea_naseer]

I'm here from the future, gpt3 can generate PDDL statements fine so it can probably generate triples in text

[@interestingamerican3100]

What is wrong with Rust?

[@soopace1486]

thank you for the interesting video

[@evilunicorn9205]

Dawn Song is a badass name

[@MachineLearningStreetTalk]

Hello 🙌

[@YannicKilcher]

Hello 👋

[@first-thoughtgiver-of-will2456]

dont bait me with the rust shoutout.

[@G12GilbertProduction]

I think this tetradecimal language structure of this model was not really tetradecimal than be are, only sextadecimal.

[@greencoder1594]

Start @ [6:44]

[@7th_CAV_Trooper]

Thumbs up for South Park reference.

[@ChaoticNeutralMatt]

Also man you've been around a while

[@bobsavage3317]

This paper makes too many assumptions. For example, "Is" and "Was" are looked up as different Relations. Also, "To Be" is often a sign that the terminal Entity is actually a property of the Subject leading to semantic statements like P(x, TRUE), e.g. the triple (Ernst, Pacifist, TRUE). Another apparent assumption is that a sentence will only have 1 fact in it. The list goes on! It's a shame, because the title suggests there could be a way to extract the semantic concept in the model and externalize them in a manner that is machine verifiable. The ability to audit a model on a given topic would be very helpful.

[@ChaoticNeutralMatt]

This feels very unexplored and unrefined. I'll have to hold onto this.

[@tinyentropy]

After half of the video, I felt it is a waste of time. Unfortunately, you didn't properly set expectations by the title of the video.

[@dr.mikeybee]

Thanks for the free 100 days.

[@marouanemaachou7875]

Keep it coming ! haha

[@dmitrysamoylenko6775]

"Jeffrey Epstein tour with" could be useful

[@mathematicalninja2756]

Love thid