A language for LLM prompt design | Guidance

Guidance calls itself "a guidance language for controlling large language models", and is an open source project by Microsoft that "enables [prompt designers] to control modern language models more effectively and efficiently than traditional prompting or chaining". This is a tutorial on Guidance to see whether it delivers on that promise and how it fits into the LLM (large language models, like GPT-3, Cohere, Alpaca, Claude etc) tooling ecosystem.
Instead of parting ways with your money on "prompt engineering gurus" only to learn how to write clear instructions in English (💩!), invest the time and money into learning how to create better interfaces between prompt and output using a framework (or more accurately templating language) like Guidance. It works with almost all LLMs (proprietary ones like GPT, to open source models on huggingface).
- Watch PART 1 of the LangChain / LLM series:
Build a GPT Q&A on your own data
• LangChain + OpenAI tut...
- Watch PART 2 of the LangChain / LLM series:
LangChain + OpenAI to chat w/ (query) own Database / CSV!
• LangChain + OpenAI to ...
- Watch PART 3 of the LangChain / LLM series
LangChain + HuggingFace's Inference API (no OpenAI credits required!)
• LangChain + HuggingFac...
- Watch PART 4 of the LangChain / LLM series
Understanding Embeddings in LLMs (ft LlamadIndex + Chroma db)
• Understanding Embeddin...
- Watch PART 5 of the LangChain / LLM series
Query any website with GPT3 and LlamaIndex
• GPT scrapes + answers ...
- Watch PART 6 of the LangChain / LLM series
Locally-hosted, offline LLM w/LlamaIndex + OPT (open source, instruction-tuning LLM)
• Locally-hosted, offlin...
- Watch PART 7 of the LangChain / LLM series
Building an AI language tutor: Pinecone + LlamaIndex + GPT-3 + BeautifulSoup
• Building an AI languag...
- Watch PART 8 of the LangChain / LLM series
Building a queryable journal 💬 w/ OpenAI, markdown & LlamaIndex 🦙
• Building a queryable j...
- Watch PART 9 of the LLM series
Making a Sci-Fi game w/ Cohere LLM + Stability.ai: Generative AI tutorial
• Making a Sci-Fi game w...
- Watch PART 10 of the LLM series
GPT builds entire app from prompt (ft. SMOL Developer)
• GPT builds entire app ...
Link to Guidance (GitHub): github.com/mic...
All the code for the LLM (large language models) series featuring GPT-3, ChatGPT, LangChain, LlamaIndex and more are on my github repository so go and ⭐ star or 🍴 fork it. Happy Coding!
github.com/onl...

[@goldenknowledge5914]

Seriously bro, you are a godsend

[@SamuelChan]

I try to be helpful! :) thank you!

[@kevingage4157]

A lot of the comments here ask about using guidance along side langchain. Now that the library has developed a bit more i would love to see a follow up video where you use guidance along with langchain to get structured output from a natural language query.

[@danieladashek5816]

Did some work with guidance today and was fairly disappointed. Ran into problems doing some simple things with it. If it remains active in a few months I will take another look. Thanks for making the video Samuel.

[@SamuelChan]

Thanks for sharing your experience! It is an early stage project ;) - if you’re inclined in helping out the development process consider filing Issues and bug reports too (or even contributing fixes) so they can be addressed! Waiting out a few months of development before giving it another go is fair too!

[@oz4549]

I remember seeing it while I was looking for ways to get structured output conditionally based on questions asked. This is mainly to render content nicely on a chatbot UI. For example, maybe I want to show clickable buttons on the chat for certain questions but for others just normal text. This means that for the first scenario my json must be an array, but in the second scenario it must be just a string. I'm not sure if it's even possible.

[@SamuelChan]

Conventional conversational UX addresses these problems using Intents, Entities and logic blocks. So for example if the user is asking for a menu, the intent is “getMenu” and now you display two buttons, English and Mandarin for example to give the user a choice of their menu language. But if the intent is “askRecommendations” then you enter a LLM chat to have back and forth chat asking for guests preferences. I used to run a chatbot engineering company and develop these custom conversational systems for companies (also have my own chatbot on Slack marketplace, was on product hunt etc) so those are from my experience in 2014-2016 - may have been outdated! Guidance and such I’d use it for the LLM output control, but I would also lean on conventional / traditional chatbot design systems for what you just described alongside modern LLM tooling

[@oz4549]

@@SamuelChan Thanks for the insight. I get your example about the languages which is cool if I have predetermined configuration like language. But say you might ask something like what are the options for X vs what does this option in X means. In the first case, I might want this to be a list of buttons but in the second case a text. All of this output is coming from query the vectordb. I just want to control how it looks. I tried something like guardrails but still lacking a bit. I will look into your suggestion. Thanks 🙏

[@SamuelChan]

Yeah so you have a case of simple intent classification at the top, classifying “what are the options of X”, “what does X option mean” etc into one of lets say 7-10 pre-defined intents. You could develop your own classification, use one hosted on huggingface, or one that is powered by Microsoft (LUIS), Meta (Wit.ai) etc, existing chatbot development tools do a very good job once you gave it enough examples. You can then test it by giving a variation of “tell me more about option X” or “elaborate please on option Y” and these service can still classify it accordingly. If the classification identifies Intent A, then render buttons, else render text through LLM etc.

[@oz4549]

@@SamuelChan Thank you very much for taking the time and answering. I will look more into this. Thanks again.

[@JhourladEstrella]

Awesome! Thanks for sharing. It's too wordy at times but you can skip a bit and all's good again.

[@SamuelChan]

Thank you! That’s something I’ll have to work on

[@kevon217]

Very interesting syntax.

[@SamuelChan]

Yeah, it’s Handlebars-inspired, a bit like Jinja if you’ve used Django or Flask in the past :)

[@jj-kneejoint77]

Yet another great video! Thanks Samuel

[@SamuelChan]

Thanks man! Appreciate it! 🙏🏼

[@jj-kneejoint77]

@@SamuelChan One quick question, would one implement a sub question query engine within Guidance to be able to control a more complex prompting or would Guidance alone be able to do what most of these query engines in llama index can do?

[@SamuelChan]

Great question. I would say Guidace and LlamaIndex occupy a vastly different space in the LLM pipeline process. LLamaIndex is very upstream-centric: the data ingestion, indexing, vector stores. Guidance is almost on the far downstream: controlling what comes out of your output. Suppose I have 50 internal docs, I would use LlamaIndex for the upstream (text preprocessing / splitter, ingestion, indexing and vectorstores). Now on the downstream, I would like to generate markdown-valid summaries of these 50 docs, or json-valid attributes from these docs, I would hence use a robust prompt design tool like Guidance to shape the outcome and interweave it with the desired properties etc.

[@jj-kneejoint77]

@@SamuelChan I see, so I would use the response of the query engine and 'pass' it to Guidance for the desired output. Thanks Samuel again, you are a divine gift my dude

[@SamuelChan]

No problem! Glad to be of service!

[@hinaaftabali6996]

00:03 Prompt design framework for language models using Microsoft's Guidance 01:44 Simple output structures like Chain of Thought have been shown to improve LLM performance 05:04 Using templates to pass variables in command prompts 06:57 Generating a sentence and specifying the number of iterations 10:33 An introduction to key commands and syntax in LLM prompt design. 12:21 Using OpenAI's GPT to generate command line examples 16:10 Generating description using a specific format 17:52 Using the hidden block structure in LLM prompts 21:02 Create a function called Pick 3 to randomly select three commands from a set of commands. 22:33 Introduction to numbering system for LLM prompt design 25:47 The Language Model guidance allows you to control your language models more efficiently and effectively. 27:15 Guidance programs allow for more robust prompt generation and ensure syntactically valid JSON format. 30:37 Using chat-based models like chat gpt for LLM prompt design 32:08 Use hugging face models instead of GPT for LLM prompt design 35:20 Understanding handlebar syntax and white space control in LLM prompts 36:52 Beginners often get confused with certain commands in LLM prompt design 40:02 Generate multiple prompts and filter the incorrect ones 41:31 You can use different models to generate JSON for prompt design Crafted by Merlin AI.

[@fernandopratama2317]

The guidance is pretty buggy at the moment. I cannot run it properly on jupyter notebook in mac computer.

[@SamuelChan]

Any error logs? What does “not run properly” means? :) It does have problems though I’ll admit, including the ones mentioned in the video, most of them I use a workaround to circumnavigate until a later patch is released.

[@wiktormigaszewski8684]

Looks cool, but I have an error with this - I have openai.error.RateLimitError and AttributeError: module 'openai' has no attribute 'api_base'. Anyway, I have OPENAI_API_KEY set, but this Guidence software just does not work.

[@wiktormigaszewski8684]

I run this in Colab (with my free openai api key) and it only displays the prompt, without doing anything

[@lochuynh6734]

Does this tool need an openai key or must pay money for any services? Thks mate

[@SamuelChan]

Nope. Guidance is an open source project, completely free!

[@m4tthias]

Thanks for your {{guidance}} !

[@SamuelChan]

{gen “you’re-welcome-message”} ! 😊

[@Arkamas9612]

Thank you for the video, it was super informative! Do you think Guidance was designed to be used as an alternative solution to LangChain or do you see use cases where they can be used in together? I'm trying to see if i can plug in Guidance to an existing langchain application but using it for just the prompt templating without generation seems like overkill

[@SamuelChan]

Hey thank you! No, I don't see Guidance replacing tools / suites like LangChain. If you think of the whole process of building a custom LLM solution as a pipeline: LlamaIndex: upstream; designed to give you lots of facilities of reading in data, from various input, and vectorizing form LangChain: midstream; designed to offer all kind of toolings, agents etc that serve as building blocks (they call these "chains") that you chain together for your LLM Guidance: downstream; what gets spit out as output Of course in practice it's not always so clearcut, plenty of smaller projects get by using LangChain for the entire upstream to downstream process, or LlamaIndex for the upstream to midstream process etc. Also libraries tend to have similar functionalities that overlap, but I can't see Guidance as an alternative to LangChain more than it is a complementary tool.

[@43seconds]

are they vim key bindings inside vscode? you're scary good with the motions!

[@SamuelChan]

I do use Vim 😬 but I disabled vim bindings for VSCode when making video lessons as they can get in the way for beginners I do have a decent speed though (>90 wpm) but I don’t that translates to coding as much

[@lifeofdean3647]

i just use with text-davici-003 but don;t use with gpt-4 ,..., can u fix that? thank you

[@SamuelChan]

Can you elaborate on the problem? Not clear to me what is it that needs fixing bro

[@airtable.architect]

I wish every programmer would watch this video

[@SamuelChan]

I’m glad you found it in the first place!

[@moon8013]

so is this how you would start to create AI agents?

[@nikhilraj3052]

Hey , sorry to disturb again. Lets say I wrote 200 sql questions and relevant queries. I want to do EDA (Exploratory data analysis like on I have a good idea what is in the text. And it would be easy for me to present it in front of my professors if i did EDA , but i wonder how? and , do you know how should i do synthetic data post segmentation. Just heard of it while being lazy to type the questions and sql code by myself but i had no idea how to implement. Could you help me again ?

[@SamuelChan]

Hey sorry if I don’t fully understand what you’re asking but say you have 200 questions and would like to test our going from query to SQL (basically natural language sql) you need to use the LLM to perform a text to sql or text to pandas translation. The sql or pandas code is then handed off to the respective engine (using a concept called “Agents”) to execute and get you your EDA answers. I cover both of these (text to sql and text to pandas) here in this video: LangChain + OpenAI to chat w/ (query) own Database / CSV! kzbin.info/www/bejne/fKuTiH2Nr8uBhqs

[@jrenothmisquith638]

Can guidance be plugged into local models as well or any other HF model thats been fine-tuned with QLoRA or PEFT ?

[@SamuelChan]

Yeah in fact Guidance own documentation highlights a lot of open source models - and some of its features work on non-GPT (non OpenAI) models only. So yeah it has great support for the big open source LLMs

[@jrenothmisquith638]

You got my sub buddy, keep up the good work!

[@SamuelChan]

Thank you! See you around!

[@jrenothmisquith638]

@@SamuelChan Hey, a follow-up. Can guidance be paired with LangChain and LlamaIndex for managing data from csv and embedding it to parse it with the prompt ?

[@SamuelChan]

Yeah sure; Guidance is actually a very light python library, you can pretty seamlessly fit it into any workflow - LangChain and LlamaIndex are “bigger” and more opinionated but they also are more concerned with the more “upstream” operations or going from raw data to embeddings / vector store.

[@earltan739]

How do we incorporate vector stores to guidance? Great content btw

[@earltan739]

I was thinking of using lang chain and generate it from there and then insert it into guidance.

[@SamuelChan]

Thank you Earl! Check out my ChromaDB or Pinecone videos on the LLM series. Guidance is just a simple library and you’d plug it into your existing LangChain / LlamaIndex Workflow without much issues! I can combine them with one of those Pinecone / ChromaDB tutorials (I have done many of them) but I fear most viewers will feel it’s redundant and a waste of time :/

[@ajmalkhan8013]

Can we do such prompt engineering for generating images

[@SamuelChan]

No not at the moment; I firmly believe we will see a (text to image prompt design )equivalent for Guidance in the coming months though with how fast this field is moving. Actually if you’re looking to do some open source work this would have been a terrific project idea too

[@ajmalkhan8013]

@@SamuelChan thanks for appreciating the idea I didnot look into it that way... & the reason is that I have no programming background

[@SamuelChan]

Yeah it’s a great time to be coming into programming and software! We’re all beginners at some point. You have a fantastic idea so that’s extra motivation to go through with it and building this as an open source project is the among the fastest ways to get employment (:

[@ajmalkhan8013]

@@SamuelChan love you bro

[@shellcatt]

I don't believe you couldn't fit this simple stuff into 8-10 minutes...

[@pftdetroit]

need a microphone, bro

[@SamuelChan]

Feel like it’s because I lack good editing skills more than the lack of mic bro. I make almost nothing off these videos so can’t justify hiring yet for what’s essentially a hobby

[@jackyliu1105]

Could you take a breath while speaking? Listening to you talk makes me feel like I'm about to suffocate.