Phi 4 on Ollama - is it REALLY better than Claude 3.5?

[@chrishayuk]

please note at 13:46, mini got the answer right, as did llama3.3 at 14:11. i did point out earlier in the video, that position 2 is a correct answer. in the flow of the video i was focused on position 3, even though i had pointed out position 2 was also correct. you've all figured this for yourselves. i'm not editting this video as it doesn't change the narrative. apologies for missing this in the flow of the video

[@Junon15]

Saved me time and heartache figuring this out the hard way. I can ask for nothing more. Thanks!

[@chrishayuk]

super glad to hear it was useful

[@patruff]

No tool calling? Qwen will they be able to?

[@RomuloMagalhaesAutoTOPO]

Wow... very good and exploratory analysis. Thank you very much.👍 You reinforce the use specific model to specific problem, for example TOC to Agents... Thanks to remember this!

[@3750gustavo]

Nemotron got on first try when responding the '254*752-3+(2-7)+5': To evaluate this expression, we need to follow the order of operations, often remembered by the acronym PEMDAS (Parentheses, Exponents, Multiplication and Division from left to right, Addition and Subtraction from left to right). Here's how to break it down: Given expression: `254 * 752 - 3 + (2 - 7) + 5` 1. **Parentheses first**: Solve the expression inside the parentheses. - `(2 - 7) = -5` 2. **Update the expression**: Replace the solved parentheses with its result. - `254 * 752 - 3 - 5 + 5` 3. **Multiplication next** (since there are no exponents): - `254 * 752 = 191008` 4. **Update the expression again**: - `191008 - 3 - 5 + 5` 5. **Finally, handle Addition and Subtraction from left to right**: - `191008 - 3 = 191005` - `191005 - 5 = 191000` - `191000 + 5 = 191005` So, the final result of the expression `254 * 752 - 3 + (2 - 7) + 5` is **191005**.

[@sgwong513]

on PHI4, I got correct answer on first try if I give hint base on what I observe from the Nemotron output: calculate 254*752-3+(2-7)+5 To evaluate this expression, you need to follow the order of operations, often remembered by the acronym PEMDAS (Parentheses, Exponents, Multiplication and Division from left to right, Addition and Subtraction from left to right) Make sure you do the multiplication correctly, break it down and calculate multiplication step by step. Make sure you sum out multiple number correctly, break it down and calculate 2 number sum and repeat.

[@JoshuaEworo]

love the video! just found your channel too, definitely earned a sub! While it doesn’t support tool call via ollama it’s dead simple to use ollama “format: json” or “format: your schema” validation to do tool calling dead simply via python. I’d suggest exploring that further in a future video.

[@pmarreck]

FYI, on Mac at least, the BoltAI GUI app will connect to locally-running Ollama and LM Studio models served by those apps

[@aaronabuusama]

have you tried structured outputs with pydantic ai?

[@jimlynch9390]

Are you sure selecting 2 is wrong?

[@patruff]

Google Fi, Phi, but when are they going to release Fo, and Fum? As an Englishman by blood please respond.

[@chrishayuk]

Hahaha, I should release the outtakes of the intro…

[@stoicescucatalin8071]

What version of OpenWebUI are you using?

[@artoke84]

in openweb ui how did you setup a dark mode? it is so useful

[@Maisonier]

Great video. Liked and subscribed

[@chrishayuk]

Awesome thank you!

[@DriftlessCryptoToo]

Bravo!!! 🎉🎉🎉

[@malllikvs]

What is your hardware configuration ?

[@alekseyburrovets4747]

in the tokenizer_config.json the fim tags are present. but in the ollama config ( application/vnd.ollama.image.template ) only chat assistant tags are present. why is that? which config to use? does the phi4 really have the fill-in-the-middle features?

[@QuizmasterLaw]

ollama run vanilj/Phi-4 Yes?

[@chrishayuk]

yep, ollama.com/vanilj/Phi-4

[@QuizmasterLaw]

@@chrishayuk thanks! liked and commented probably subscribed better check n b sure

[@chrishayuk]

thank you, glad you found the vid useful

[@thegrumpydeveloper]

I think Sam Altman said “no one wants last years model” or the second best model. Unless it’s really faster or better most will stick to one of the others. Good to see though.

[@Tsardoz]

You dont explain how to set up openwebui with phi4 unless it is already installed. I am having all sorts of trouble with that. Openwebui does not see phi4 at all even though it is running under ollama. PS. The issue I had was with docker under Linux not wanting to access localhost. I installed it using a venv and pip. Also ollama pull vanilj/Phi-4:Q8_0 ollama serve (probably not necessary - main thing was bloody docker)

[@Corteum]

If you're really good at using the model, could get a lot of work done with it, or does it have too many limitations and weaknesses?

[@QuizmasterLaw]

not yet available on huggingface library searching but if someone has a pull command from ollama please say

[@thomecq]

ollama run vanilj/Phi-4:Q8_0

[@jacquesdupontd]

uh ? ollama run vanilj/Phi-4:Q8_0

[@henry-js]

What are your pc specs??

[@chrishayuk]

macbook pro m3 max with 128GB of unified memory

[@seniormcyt5552]

what's 254*752-3+(2-7)+5? first convert it to simpler parts then start solve it This model is very good at CoT, so if you make it think step by step, it gives you the right answer. For a better test of this model, you should make it break it down and think about it as much as possible. so the System Prompt should be : always break down any questions into simpler parts, then solve it step by step.

[@chrishayuk]

i shouldn't need to specify step by step for math problems... modern models know they need to do that for themselves.. so i purposely don't do it

[@DrWaldonHendricks]

I used a Gen10 A2 GPU, and it actually did a really good job on the latest model. It used about 10GB NVRAM at most compared with the o1, and it was not far of a really good model

[@alelondon23]

13:46 GPT4o mini got a RIGHT answer. position 2 is a winning move as much as 3 (you said it yourself earlier in the video and you mistakenly discarded the GPT4o mini answer as invalid. Again on 14:11 you wrongly stated Llama 3.3 gets it wrong when it gets it right! You had 1 job! Please fix the video or carry your reputation as a shill.

[@chrishayuk]

lol, you're correct, i got it wrong....

[@chrishayuk]

i've put a pinned comment pointing it out

The Ollama modal is quantized to Q4_K_M, so you will lose much quality, especially context. It is not a fair comparison. Also, you may need some guidance to force thinking which o1 and Claude 3.5 are doing but Ollama doesn't support yet.

[@chrishayuk]

it's not a highly quanitized model, and i'm purposely focusing on areas where the quantization won't massively effect it. the model is not suddenly gonna gain more personality (this is a model size thing rather than quantizing), similar for the chains of thought, and similar for the code issues

@@chrishayuk Q4 usually causes 5 to 10% drop in accuracy. Sometimes it can be up to 20%. Also, I said before, Ollama do not support forced thinking yet, so it is not possible to compare Phi-4 to Cladue 3.5 this way.

[@d.d.z.]

Nice video Chris

[@chrishayuk]

thaaaank you

[@FalconStudioWin]

The 14b parameter model may give worse answers in test time compute as smaller models generally does not reason better vs larger models

[@husanaaulia4717]

At this point, Supernova Medius is better?🤔

[@tnypxl]

The answer is always no.

[@chrishayuk]

yep, it always is

[@obinnaokafor6252]

Models from Microsoft are really good

[@chrishayuk]

yeah, it's a good model, just frustrating

[@obinnaokafor6252]

@chrishayuk frustrating in what sense?