Why Structured Outputs by OpenAI Change Everything

[@nickbobrowski]

We’re entering a new era of automation, shifting from an imperative approach - where every step had to be explicitly coded - to a declarative one focused on defining goals and outcomes. AI can now produce structured outputs with high accuracy, allowing for more abstract instructions. This marks a fundamental change in automation, where AI agents, equipped with the right tools and capable of understanding, planning, and acting autonomously, can complete tasks with minimal human input.

[@merlinrabens]

Well said Sir 👏🏼

[@ryzikx]

polymorphic applications, as said by david shapiro

[@digitalsoultech]

In theory that's what agents were supposed to do from beginning. But always ran into problems. We will probably run into other problems now.

[@goldenboy0527]

Good bot

[@buzzmorley6333]

Well said!

[@newfrontiers5673]

What do you think of Anthropic's new prompt caching feature?

[@coachdennis6139]

I found this in the midst of implementing the structure on one of my workflows thank you

[@jeffsogolov4750]

Really need a beginner to advanced course. Zero to hero

[@vrsen]

Got it

[@Kevinsmithns]

Hi bro i was just thinking about you and then u posted a video lol Im wondering if you can do a tutorial on using vapi with agency swarm to completely control a business including sending emails and follow up calls and also scheduling employees for specific jobs to go to a location and then managing them. Im trying to setup for an Hvac business and I want it to take a call, find out what's needed, send an invoice and follow up with the customer and then if they go with the job to organize with the employees to go to the job on a scheduled time and day and then manage everything. Do u think that's possible?

[@vrsen]

Yes I think that's possible if you deploy your agents as an API. Then, you can connect to Vapi

[@dawid_dahl]

Nice video graphics man! 🙂🙌🏻

[@SiCrip09]

What if the prompt has missing Information like location in the schema when the prompt is: „What‘s the weather?“ Does it ask for the location to fulfill the schema?

[@vrsen]

That’s a really good question, I need to check

[@simonetheditch809]

Please make a video talk about "Agency Chart".

[@MaxForbang]

Another banger from the man himself

[@vrsen]

Thanks, will do more of this format

[@josephc8440]

great video thanks!

[@AhmedMekallach]

This was possible though before this. They just made it easier for developers, which is great. But then you're stuck with the OpenAI base model... No privacy... Building your own way for 100% JSON output is the way to go, and it's relatively easy. And if you have a large nested schema, why not just break it down? It's golden. I've done millions of calls, with a 100% success rate on JSON output structure.

[@vrsen]

Their blog post has a great example of a tool for creating HTML files, where a nested schema makes sense due to the hierarchy of elements. Attempting to generate content 10 levels deep will certainly lead to hallucinations without this feature. I'll release a video next, where I build a similar React developer agent with a nested tool for creating component trees.

[@ew3995]

this is just a wrapper around the instructor library, but worse because you cant write validation code

[@vrsen]

No, it’s different. Watch till the end

[@konstantinlozev2272]

They are 100% correct in output form. Not 100% in output content.

[@DihelsonMendonca]

💥 YOU NEED to check your subtitles, because your English is not native, abd you say one thing, and the subtitles are a little bit different. 🙏👍

[@digitalsoultech]

Literally useless feature if they can't assure privacy. All business need privacy built in.

[@vrsen]

That’s true. Hopefully it’s coming to azure soon

[@Aryankingz]

OpenAI has an Enterprise version for this -- you would ned to contact their sales department for details and access to this

[@digitalsoultech]

@@Aryankingz in that case better to use llama 3.1

[@lucassaccone]

Is this only through their official API or can it be used with langchain? So far I've been using the output parsers with pydantic in langchain with great results

[@Skhulile84]

Thanks for continually maintaining your framework! Can't wait to try this new structured output

[@vrsen]

Install from git for now, I wanna add a few things before the next release

[@MrVohveli]

I can finally use AI inside code in a way where the output is actionable by traditional code. It can literally be a part of the program now and it's so awesome I almost shed a tear when I read this.

[@SouhailEntertainment]

Introduction and Overview of Structured Outputs by OpenAI - 00:00:00 The State of AI Adoption in 2024 - 00:00:37 Importance of Function Calling for AI Agents - 00:01:16 Challenges of Traditional Automation vs. AI Agents - 00:02:31 Understanding JSON Schemas in AI Agents - 00:03:05 Introduction to Structured Outputs by OpenAI - 00:04:10 How Structured Outputs Ensure 100% Accuracy - 00:05:20 Usage and Limitations of Structured Outputs - 00:07:31 Demonstration: Using Structured Outputs with Function Calling - 00:08:03 Demonstration: Using Structured Outputs with Response Format - 00:09:09 Comparison with Instructor Library - 00:10:46 Building AI Agents Using Structured Outputs in the Agency Framework - 00:11:21 Conclusion and Final Thoughts - 00:13:21

[@Canna_Science_and_Technology]

Don’t forget that this is structured output. You can use the structured output in a function the way you need that function to use it. But don’t lose focus on the structured output. For example, you can create a real time user interface that can adjust or be modify based on user input so basically a. Single Paged Website that changes based on the LLM’s response. It actually is pretty cool.

[@brianWreaves]

One step closer to personalised digital products for everyone.

[@vrsen]

That’s a really cool idea! Can you try this and share on our discord?

[@yurijmikhassiak7342]

That was the missing peace. Now you can get reliable outputs for simple ui components and sql queries. But it’s slower than hardcoded components.

[@rajeshjsl]

@@yurijmikhassiak7342not for long

[@danielmartinezdev]

Hey @vrsen do you know if structured outputs is available in the Azure OpenAI SDK yet?

[@acanizares_]

Is possible to maintain the context with assistants and still using structured outputs via API?

[@yuichi.inumaru]

Man you are legendary

[@RaviKumarVerma-j7w]

This is very good functionality , I myself faced getting structured output in production

[@ManasSharma-e4m]

As cs first year scares me what i do for future

[@Nuninecko]

thanks for another great video, just my opinion on something: not hallucination, the problem is working with tools EXACTLY like humans do

[@vrsen]

That’s what figure is doing. Tools that control a robot body 🤖

[@Nuninecko]

​@@vrsenvirtual or physical, same point I am making. We have LLMs speaking our language and instead of focusing on ONE thing only,ie, access and do everything just like we do, the way we access data etc, we are yet still trying to speak the computer language, getting APIs and tools in the AI platforms for it, instead of focusing on that ONE thing, access everything online/PC etc like a human. AI is so fast that it does not matter if it accesses everything our way. And it is better to only do that, as billions of online/apps/platforms on our machines and internet are made for us already. Just my view...

[@boyardosalmon]

How about the data being complete? Many times it just outputs “samples”, so it’s structured JSON but incomplete. Is there a solution for that?

[@vrsen]

Yeah, try setting default values

[@quantanovabusiness]

when are you doing a tutorial with your software working on open sourced LLM's, what if we dont want to use openAI?

[@nickbobrowski]

it was already posted recently on David Ondrej’s channel, Agency Swarm with Llama 3.1

[@Jaun_]

I am a big advocate for structured output - let’s content generation, more structure and functional use.

[@vrsen]

Yeah, helps you save on tokens if used right as well

[@AlexanderShelestov]

Почему мне кажется, что вы уже сами говорите как АИ? :)

[@vrsen]

Возможно, вам кажется, что я говорю как ИИ, потому что моя речь часто структурирована, логична и лишена эмоциональной окраски, которую мы привыкли видеть в общении с людьми. Это связано с тем, что я стараюсь предоставлять точную, полезную информацию и избегать двусмысленностей, что может создавать впечатление определённой "неестественности" общения. Кроме того, я обучен анализировать и генерировать текст на основе огромного количества данных, что делает мою речь более формальной и прямолинейной. Если вам кажется, что моя речь слишком машинная или формальная, я могу адаптировать стиль общения, чтобы сделать его более естественным и живым, если это вам будет удобнее.

[@AlexanderShelestov]

@@vrsen ​ Хаха, даже комент звучит как ИИ :) Не, мне все норм, ничего менять не надо, если это ваш стиль и вам так удобнее доносить до публики ваши идеи. Я, кстати, неделю назад еще не понимал эту тему от слова совсем. Но посмотрел ваши ролики и других ребят и на выходных уже сделал PoC бота технической поддержки в своей компании, с RAG, qdrant и Claude агентами. Писал на Ruby, но думаю сделать полноценный проект на Elixir/Phoenix.

[@AI_Escaped]

@@vrsen LOL :)

[@AI_Escaped]

​@vrsen anyone else that works with AI 12 hours a day starting to think like generative AI? I find myself thinking like I'm generating responses to my other own prompts :)

[@AlexanderShelestov]

@@AI_Escaped haha, so true.

[@brianWreaves]

This is an impressive achievement!

[@vrsen]

Yes, definitely deserves more attention