AI Weekly Update - January 24th, 2022

[@TimScarfe]

Great to see these back!

[@connor-shorten]

Thanks Tim, appreciate it!

[@domenickmifsud]

Thanks!

[@connor-shorten]

You're welcome! Thanks for watching!

[@billykotsos4642]

Weviates technology looks REALLY interesting. Do they have something akin to a white paper covering the technology?

[@connor-shorten]

Thanks Billy, really happy to hear that! Maybe Bob will drop in the comments soon to give you what they recommend -- I'd suggest just checking out the Weaviate documentation, scrolling down to Tutorials, and looking at "How to query data?". I think that will provide a lot of inspiration for the GraphQL API for Neurosymbolic search. More than happy to answer any more questions you have about this, also please check out the Weaviate slack, great community for these kinds of discussions!

[@billykotsos4642]

@@connor-shorten Oh thanks ! nice !

[@SantoshGupta-jn1wn]

re promptBert The paper gave explanations on why it outperforms the original Bert, but they also compare it to SOTA model that fine-tune the whole model with using either the CLS token or average pooling and promptBert outperforms them, but it doesn't seem to give a solid explanation why. Any idea?

[@connor-shorten]

They probably don't want to give away too many of the secrets of it haha, but generally this idea of prompting large language models seems like the future of many applications --- the explanation being that the contextual representations are so powerful that we have been misevaluating these systems this entire time with a lack of proper context in testing

[@SantoshGupta-jn1wn]

​@@connor-shorten Kinda mind blowing. It definitely makes sense with limited data. But I would imagine fine tuning with just the CLS token would be all the beneficial nudging a model can get for the task, with enough data; the attention heads and linear layers should get enough information to pinpoint the task. But the prompting also outperformed on the unsupervised models (I'm guessing the unsupervised training had large amounts of data). So maybe having some input tokens dedicated to storing task information frees up the attention heads / linear layers to keep focused on their original purpose. Maybe attention heads are not well suited for tasks not directly involved with token-to-token attention (ie anything outside of token classification)

[@connor-shorten]

@@SantoshGupta-jn1wn I agree with all the conclusions in this post -- w.r.t. fine-tuning, you might be interested in these techniques that do continuous prompt tuning, no reason the prompt needs to be an actual text sequence if you are going to optimize it with gradients

[@InquilineKea]

Wait Ray kurzweil is on a paper.???

[@connor-shorten]

Haha, maybe I don't really study the authors names -- which one did you see?