PAL : Program-aided Language Models with LangChain code

[@AndiEliot]

Please note PALChain has now moved to langchain_experimental. Also code-davinci-002 model does not work use gpt-3.5-turbo instead.

[@samwitteveenai]

Yes this video was made over 6 months ago and they have changed, gpt-3.5-turbo is also not the best at PAL. GPT-4 works better but some of the open source models have also been getting better lately too.

[@mitchellscott2992]

Fantastic videos Sam - this playlist has (happily) consumed my entire night. Who needs fresh laundry anyway?!

[@samwitteveenai]

Hopefully the LLMs can do the laundry soon. :D Thanks for the kind words.

[@GeoffY2020]

Fantastic video ! I watched it like 3 times over last few days as it makes me think what to append to make LLM useful ... feed us more ! thanks

[@samwitteveenai]

Glad you enjoyed it! There are a few more papers this with cool examples of using prompting to do unconventional things. You might like the video and code on Visual ChatGPT which uses other external tools via prompting as well.

[@MrAkhilanil]

Hey great video, what did you mean when you said the example with reaching to school is incorrect because there were no similar prompts before. I can see it followed the exact same step as the first example.

[@bandui4021]

What would be the approach to find the issue why it is calculating wrongly? Is there a way to look into the NN or we are just changing the prompt and hope for a better result. So my question is how we could "look" into the NN to find the error or i am thinking to easy? Thanks

[@samwitteveenai]

probing in the NeuralNet it's self is a major research effort. Most likely if it is the model itself that is calculating wrongly, then it just hasn't seen enough examples. If the issue is with PAL then I would suggest you work on modifying the prompt.

[@goelnikhils]

Very good video on Program-Aided Large Language Models

[@venkatesanr9455]

Thanks for your video, Sam. I hav one doubt whether langchain+LLMs like flan T5 x, T5 xl, flan 20 B & flan UL20B accessing using huggingface tokens ( other than downloading model) is free of service. When we are considering alternative to open ai gpt3 models.

[@samwitteveenai]

Their API is is not a totally free of service and to be honest I wouldn't use the free version to put something into production, that said I have never maxed out my free API calls so far. There doesn't seem to be a lot of info on their pricing page huggingface.co/pricing this could be that they are in startup mode to get lots of users before charging etc. The main disadvantage I have seen is that some of the bigger models often don't work with the API. I do wonder how long they will keep the Flan UL20B up for free for example. If I find something better I will certainly make a video of it. If you find something else please let me know :D

[@nicolasmenescardi4077]

great video! Is OpenAI function calling replacing this?

[@samwitteveenai]

they can if used with REPL etc but function calling has more replaced ReACT etc.

[@scottstults8756]

In March of 2023 OpenAI deprecated the code-davinci-002 model. You can get the first example to run if you replace it with "gpt-3.5-turbo".

[@aayushsmarten]

Its good but I have a case where the model has to "think through by calculating in between the steps" doing this will make many back and forth calls to the model and the interpreter. So when the interpreter returns the result back to the model, it needs to start from scratch. Unless we have some "continuation" models... there should be some way to "pause" the model and when the answer comes from the interpreter, the model should resume its generation.

[@ratral]

Sam, When I first saw the video, the PAL module's importance was unclear to me (mea culpa). Now watching the video again and reading the article, I recognize great potential. Thanks for the clarification.

[@samwitteveenai]

unfortunately that model is no longer publicly available on OpenAI.

[@jakubbartczuk3956]

Hey man, great video. Did you read LMQL paper by any chance?

[@samwitteveenai]

I did its certainly an interesting idea. I might make a vid on it.

[@1peacockeye]

Thank you very much for your fantastic videos! Can a PAL also be simulated within a LLM? So, through specific prompting the LLM simulates a Python environment in which it does the calculation and then returns the answer? Does that make sense?

[@samwitteveenai]

It has been tried and can work to a degree, but often will get small parts wrong and thus hasn't been reliable so far.

[@henkhbit5748]

Nice and interesting if u want to aks more math question, you used code-davinci-02 in the setup for pal but can u combine it with the normal llm like chatgpt-turbo3.5 ?

[@samwitteveenai]

Yes but it doesn't always work as well. unfortunately the code-002 model has now been taken down on the OpenAI API. GPT-4 does a pretty good job

[@florianstephan5745]

Hey Sam, great Channel, appreciate it!!!!

[@jacktherater3533]

Makes sense. Even I can't do math without calculator.

[@shouldb.studying4670]

+

[@daniel_grindforce994]

Great job man. I subbed. Lets keep em coming!

[@ObservingBeauty]

Great video. Very helpful

[@samwitteveenai]

Glad it was helpful!