Structured Outputs with DSPy

[@connor-shorten]

Hey everyone! I received some great notes from Thomas Ahle, the creator of DSPy's TypedPredictors. These have been added to the video description, but going to leave them here as a pinned comment as well to make them easier to find! **ERRATA** 1. `dspy.TypedPredictor` can be used directly instead of `dspy.functional.TypedPredictor` 2. When creating a pydantic type, `list[Topic]` can be used directly in the Signature without needing the `Topics` wrapper. 3. The default `max_retry` for TypedPredictor is 3, and can be set when creating the TypedPredictor. 4. Setting `TypedPredictor(explain_errors=True)` can help with retry errors by providing clearer descriptions of what needs to change.

[@ecardenas300]

Awesome video, Connor! Excited to try out the TypedPredictors with assertions

[@connor-shorten]

Thanks so much Erika! Yes! TypedPredictors and Assertions are the Peanut Butter and Jelly of LLM systems!

[@Dan-e6r7s]

Thanks for sharing this! I'd love to see you doing something with the ReAct module, especially optimizing it using DSPy.

[@connor-shorten]

Awesome idea! Definitely looking forward to learning more about function calling with DSPy and the ReAct framework for doing so!

[@TongLi-g2f]

Amazing guide! Was trying to format llama-3-8B for a while and this provides so much depth to help me understand how to go about it.

[@parttimelarry]

Thanks for the video, just starting to wrap my head around these techniques and your (and your colleague's) videos have been helpful. Cheers.

[@connor-shorten]

Thank you so much! Means a lot to hear that! Cheers!

[@EberLaurente]

I'm very grateful for this videos series. I'm learning a lot.

[@connor-shorten]

Thank you so much! So happy to hear it!!

[@savannahquire5414]

Wow this video blew my mind!! I didn’t know any of this before, this was super helpful Dr. Connor!

[@connor-shorten]

Fantastic! Thank you so much!

[@mysticaltech]

Just FYI people, about models for function calling and returning JSON, Hermes Pro 2 7B from NousResearch is also working extremely well, and was trained with those use cases in mind.

[@connor-shorten]

Epic! Going to check it out, thanks so much for the tip!

[@YuryGurevich]

Great content, well presented! Thanks for the whole series. I have two questions: What are the fundamental differences between a retrieval model and a "main" LLM? I know the idea of DSPY is to hide the LLM from coding choices. In the context of LLMs, what are the teacher/student scenarios used for? What actually happens when one compiles a DSPY component? I'd appreciate a deep dive into this topic.

[@AyaAya-fh2wx]

Hey Connor, thanks so much for your work

[@rkenne1391]

Thanks for the great content. How will you combine instructor with DSPY ? Instructor seems to have resolve structured output in a more elegant way. I like fully embracing framework but Instructor have an almost invisible abstractions for addressing structured outpout

[@ludwigstumpp]

Hi Connor, thanks the awesome video! This serves as a great intoduction to using DSPy for Constrained Output Generation Tasks! I was wondering if there is any way instead of using the default JSON functionality of TypedPredictor as a way to tell the model the correct format to also use the Tool/Function call functionality that many LLM APIs provide -> so in a sense similar to what the instructor package allows. If there is not yet a native way doing this in DSPy, is there another possibility to use DSPy togeter with the instructor library? What I would love to do is to use DSPy on top of instructor to optimize over the instruction part of the prompt (if we imagine prompt = constant instruction + changing input) that is sent to the LLM via instructor. The current only workaround I found is to use the DSPy integration of Langchain as Langchain also offers constrained output generation via Tool/Function call functionality. But maybe there is a better alternative that would allow me not to use Langchain (as very much bloated up). Any hints or directions would be very much appreciated! Thanks a lot for your many DSPy tutorials, this is simply awesome!

[@artur50]

is there going to be any GUI implemented on top of that?

[@connor-shorten]

Love this type of thinking!! I have a project on github -> weaviate-tutorials -> Hurricane, definitely more to come! Arize Phoenix gives you a nice UI for program traces if that is what you mean with this question instead.