Building a Vision App with Ollama Structured Outputs

[@zhalberd]

This channel is straight up priceless. No fluff. Real deal development how-to. Thank you

[@samwitteveenai]

Thanks it's appreciated.

[@davidmccauley7822]

I would love to see a simple example of how to fine-tune a vision model with ollama.

[@suiteyousir]

Thanks for these updates, quite difficult to keep up with all the new releases nowadays

[@chizzlemo3094]

OMG, this is exactly what I need. Thanks so much.

[@bigfootpegrande]

Miles and IA? I'm all for it!

[@gr8tbigtreehugger]

Cool to see how you approached NER using an LLM. I've been using SpaCy.

[@samwitteveenai]

I normally use Spacy for anything at scale. You can use LLMs to make good datasets for custom entities and then use that to train the Spacy model

[@NeuralDev]

Could you make a more in depth tutorial about finetuning model to improve their accuracy for this type of task ?

[@loudmanCA]

Really appreciate your channel! Could you make a video to help us better understand what specs are required for using LLMs locally?

[@sridharangopal]

Great videos, Sam. Learnt so much from your videos. RE: Llama vision model on Ollama, I have been trying to get it to work with both pictures and tools but it looks like it can only do pictures and structured output and no tool calling support yet. Any idea on how to get around this limitation?

[@marouahamdi4293]

I love this video as always! I have several invoices from which I want to extract the information and save it into an Excel file. I imagine this is doable with this structured output technique. If you have any advice on how to do it, I’m all ears!

[@samwitteveenai]

Extract it as JSON and then use something like Pandas or openpyxl to save it to excel.

[@Stewz66]

So, is intelligent document processing and document classification possible with open source vision models ??? Wheels turning...

[@samwitteveenai]

Yes, but it definitely helps if you fine-tune the model for your particular use case.

[@justine_chang39]

do you know if this model would be good for getting the coordinates of stuff in images? For example I would like to get the coordinates of a dog in an image, the model might return a bounding box [[x1, y1], [x2, y2]]

[@samwitteveenai]

These models are probably not good enough for that at the moment, but certainly things like the new Gemini model can do that kind of task.

[@parnapratimmitra6533]

Very informative video regarding Vision based models with structured outputs. If possible, could you also make a video on a simple langchain or langgraph app using vision based models of ollama for reading and describing into structured outputs, all the images in a document let's say pdf? Thanks in advance

[@protovici1476]

Check out ColPaLi.

[@austinlinco]

I literally thought of this yesterday and was using a system prompt to force it to respond as a dictionary Wtf is up with 2025 being perfect, and what’s the catch

[@nufh]

I have tried it, it depends on the model itself.

[@PandoraBox1943]

very useful

[@pensiveintrovert4318]

The amount of hacking you have to do to just get "ok" results says it all. Not production quality, and won't be any time soon.

[@brando2818]

Have you tried it with better models than were used here?

[@adriangabriel3219]

that's not to be expected with models of that size

[@pensiveintrovert4318]

@@brando2818 the whole point of using Ollama is to run open source models, on your own hardware. OpenAI, Anthrop\c, Google already offer structured output.

[@RohitSharma-uw2eh]

Why Hindi audio track is not available 😢

[@samwitteveenai]

How do I turn this on? Will look into it.

[@surfaceoftheoesj]

very useful