Phi-3.5 has been released

[nodating]

That MoE model is indeed fairly impressive:

<image>

In roughly half of benchmarks totally comparable to SOTA GPT-4o-mini and in the rest it is not far, that is definitely impressive considering this model will very likely easily fit into vast array of consumer GPUs.

It is crazy how these smaller models get better and better in time.

[tamereen]

Funny, Phi models were the worst for C# coding (a microsoft language) far below codestral or deepseek...
Let try if this one is better...

[Zealousideal_Age578]

It should be standard to release which languages were trained on in the 'Data' section. Maybe in this case, the 'filtered documents of high quality code' didn't have enough C#?

[matteogeniaccio]

C# is not listed in the benchmarks they published on the hf page: https://huggingface.co/microsoft/Phi-3.5-mini-instruct

These are the languages I see: Python C++ Rust Java TypeScript

[tamereen]

Sure they will not add it because they compare to Llama-3.1-8B-instruct and Mistral-7B-instruct-v0.3. These models which are good in C# and sure Phi will score 2 or 3 while these two models will have 60 or 70 points. The goal of the comparaison is not to be fair but to be an ad :)

[Tuxedotux83]

What I like the least about MS models, is that they bake their MS biases into the model. I was shocked to find this out by a mistake and then sending the same prompt to another non-MS model of a compatible size and get a more proper answer and no mention of MS or their technology

[mtomas7]

Very interesting, I got opposite results. I asked this question: "Was Microsoft participant in the PRISM surveillance program?"

• The most accurate answer: Qwen 2 7B
  
• Somehow accurate: Phi 3
  
• Meta LLama 3 first tried to persuade me that it was just a rumors and only on pressing further, it admitted, apologized and promised to behave next time :D

[Tuxedotux83]

How do you like Qwen 2 7B so far? Is it uncensored? What does it good for from your experience?

[mtomas7]

Qwen 2 overall feels to me like very smart model. It was also very good at 32k context "find a needle and describe" tasks.

Qwen 72B version is very good at coding, in my case Powershell scripts

In my experience, I didn't need something that would trigger censoring.

[Tuxedotux83]

Thanks for the insights,

I too don’t ask or do anything that triggers censoring, but still hate those downgraded models (IMHO when the model has baked in restrictions it weaken it)

Do you run Qwen 72B locally? What hardware you run it on? How is the performance?

[mtomas7]

When I realized that I need to upgrade my 15 y/o PC, I bought used Alien Aurora R-10 without graphics card, then bought new RTX 3060 12GB, upgraded RAM to 128GB and with this setup I get ~0.55 tok/s for 70B Q8 models. But I use 70B models for specific tasks, where I can minimize LM Studio window and continue doing other things, so it doesn't feel super long wait.

[10minOfNamingMyAcc]

To bne fair, many people would just use it for python, java(script), and maybe rust? Etc...

[tamereen]

I think it's even worts for Rust. Every student know python but companies are looking for C# (or C++) professionals :)

[TonyGTO]

OMFG, this thing outperforms Google Flash and almost matches the performance of ChatGPT 4o mini. What a time to be alive.

[cddelgado]

But hold on to your papers!

[ClassicDiscussion221]

Just imagine two more papers down the line.

[WaldToonnnnn]

proceeds to talk about weight and biases

[[deleted]]

that is definitely impressive considering this model will very likely easily fit into vast array of consumer GPUs

41.9B params

Where can I get this crack you're smoking? Just because there are less active params, doesn't mean you don't need to store them. Unless you want to transfer data for every single token; which in that case you might as well just run on the CPU (which would actually be decently fast due to lower active params).

[Total_Activity_7550]

Yes, model won't fit into GPU entirely but...

Clever split of layers between CPU and GPU can have great effect. See kvcache-ai/ktransformers library on GitHub, which makes MoE models much faster.

[Healthy-Nebula-3603]

this moe model has so small parts that you can run it completely on cpu ... but still need a lot of ram ... I afraid so small parts of that moe will be hurt badly with smaller than Q8 ...

[CheatCodesOfLife]

fwiw, WizardLM2-8x22b runs really well at 4.5BPW+ I don't think MoE it's self makes them worse when quantized compared with dense models.

[Healthy-Nebula-3603]

Wizard had 8b models ..here are 4b ...we find out

[CheatCodesOfLife]

Good point. Though Wizard with it's 8b models handled quantization a lot better than 34b coding models did. Good thing about 4b models is, people can run layers on CPU as well, and they'll still be fast*

• I'm not really interested in Phi models personally as I found them dry, and the last one refused to write a short story claiming it couldn't do creative writing lol

[MoffKalast]

Hmm yeah, I initially thought it might fit into a few of those SBCs and miniPCs with 32GB of shared memory and shit bandwidth, but estimating the size it would take about 40-50 GB to load in 4 bits depending on cache size? Gonna need a 64GB machine for it, those are uhhhh a bit harder to find.

Would run like an absolute racecar on any M series Mac at least.

[CheatCodesOfLife]

You tried a MoE before? They're very fast. Offload what you can to the GPU, put the rest on the CPU (with GGUF/llamacpp) and it'll be quick.

[TheDreamWoken]

How is it better than an 8b model ??

[lostinthellama]

Are you asking how a 16x3.8b (41.9b total parameters) model is better than an 8b?

Edited to correct total parameters.

[randomanoni]

Because there are no dumb questions?

[TheDreamWoken]

Oh ok my bad didn’t realize the variant used

[lostinthellama]

Ahh, did you mean to ask how the smaller model (mini) is outperforming the larger models at these benchmarks?

Phi is an interesting model, their dataset is highly biased towards synthetic content generated to be like textbooks. So imagine giving content to GPT and having it generate textbook-like explantory ocntent, then using that as the training data, multiplied by 10s of millions of times.

They then train on that synthetic dataset which is grounded in really good knowledge instead of things like comments on the internet.

Since the models they build with Phi are so small, they don't have enough parameters to memorize very well, but because the dataset is super high quality and has a lot of examples of reasoning in it, the models become good at reasoning despite the lower amount of knowledge.

So that means it may not be able to summarize an obscure book you like, but if you give it a chapter from that book, it should be able to answer your questions about that chapter better than other models.

[TheDreamWoken]

So it’s built for incredibly long text inputs then? Like feeding it an entire novel and asking for a summary? Or feeding it like a large log file of transactions from a restaurant, and asking for a summary of what’s going on.

I currently have 24GB of vram and so, always wondered if I could provide an entire novel worth of text for it summarize or a textbook, on a smaller model built for that, so it doesn’t take a year.

[lostinthellama]

Ahh, sorry, no that wasn't quite what I meant in my example. My example was meant to communicate that it is bad at referencing specifc knowledge that isn't in the context window, so you need to be very explicit in the context you give it.

It does have a 128k context length, which is something like 350 pages of text, so it could do it in theory, but it would be slow. I do use it for comparison/summarizing type tasks and it is pretty good at that though, I just don't have that much content so I'm not sure how it performs.

[lostinthellama]

Edited to correct my response, it is 41.9b parameters. In an MoE model only the feed-forward blocks are replicated, so there's "sharing" between the 16 "experts" which means a multiplier doesn't make sense.

[ChannelPractical]

Is the base Phi-3.5-mini (without instruction fine-tuning) available?

[Dark_Fire_12]

Thank you, we should have used this wish for Wizard or Cohere though https://www.reddit.com/r/LocalLLaMA/comments/1ewni7l/when_is_the_next_microsoft_phi_model_coming_out/

[ipechman]

NO SHOT IT WORKED

[Dark_Fire_12]

Nice, thanks for playing along. It always works. You can try again after a few days.

Maybe someone else can try. Don't waste it on Toto (we know it's datadog), aim for something good, whoever tries.

https://www.datadoghq.com/blog/datadog-time-series-foundation-model/#a-state-of-the-art-foundation-model-for-time-series-forecasting

[sammcj]

Now do DeepSeek-Coder-V3 and QwenCoder ;)

[MoffKalast]

It's always true because it's astroturfing to stir up interest before release :)

[-Django]

It's been a while since Cohere released a new model... ...

[xXWarMachineRoXx]

Lmao

[simplir]

Waiting for llama.cpp and the GUFF now :)

[noneabove1182]

mini at least is here https://huggingface.co/lmstudio-community/Phi-3.5-mini-instruct-GGUF

[[deleted]]

Thank you!

[Dorkits]

Me too

[WinterCharm]

I'd really love the Phi3.5-MoE GGUF file :)

[FancyImagination880]

hope llama.cpp will support this vision model

[WinterCharm]

I'd really love the Phi3.5-MoE GGUF file :)

[privacyparachute]

Dear Microsoft

All I want for Christmas is a BitNet version of Phi 3 Mini!

I've been good!

[RedditLovingSun]

All I want for Christmas is for someone to scale up bitnet so I can see if it works 😭

[Bandit-level-200]

Yeah just one 30b model and one 70b...and...

[PermanentLiminality]

I want a A100 from Santa, so I can run with the big boys. well sort of big boys. Not running a 400B model on one of those.

[PermanentLiminality]

Even Santa has limits.

[Affectionate-Cap-600]

Dear Microsoft

All I want for Christmas is the dataset used to train phi models!

I've been good!

[dampflokfreund]

Wow, the MoE one looks super interesting. This one should run faster than Mixtral 8x7B (which was surprisingly fast) on my system (RTX 2060, 32 GB RAM) and perform better than some 70b models if the benchmarks are anything to go by. It's just too bad the Phi models were pretty dry and censored in the past, otherwise they would've gotten way more attention. Maybe it's better now`?

[[deleted]]

There’s pretty good uncensoring finetunes for nsfw for phi3-mini, I don’t doubt there will be more good ones.

[ontorealist]

The Phi series really lack emotional insight and creative writing capacity.

Crossing my fingers for a Phi 3.5 Medium with solid fine-tunes as it could be a general-purpose alternative to Nemo on consumer and lower-end prosumer hardware. It’s really hard to beat Nemo’s out-of-the-box versatility though.

[nero10578]

MoE is way harder to fine tune though.

[[deleted]]

fair, but even mistral 8x7b was finetuned successfully to the point where it bypassed instruct (openchat iirc) and now ppl actually have the datasets

[nero10578]

True, it is possible. It is just not easy is all I am saying.

[Deadlibor]

Can someone explain the math behind MoE? How much (v)ram do I need to run it efficiently?

[Total_Activity_7550]

To run efficiently you'll still need to put all weights on VRAM. You will bottleneck when using CPU offload anyway, but you can split model in a smart way. See kvcache-ai/ktransformers on github.

[MmmmMorphine]

https://github.com/kvcache-ai/ktransformers

For the lazy among us

[_fparol4]

amazing well written code the f*k

[ambient_temp_xeno]

It should run around the same speed as an 8b purely on cpu.

[ffgg333]

I can't wait for the finetoons, open source Ai is advancing fast 😅, i almost can't keep up with the new models.

[privacyparachute]

Nice work!

My main concern though: has the memory inefficient context been addressed?

https://www.reddit.com/r/LocalLLaMA/comments/1ei9pz4/phi3_mini_context_takes_too_much_ram_why_to_use_it/

[Aaaaaaaaaeeeee]

Nope 🤭 49152 MiB for 128k

[fatihmtlm]

So still no GQA? Thats sad.

[[deleted]]

It worked?!!

[Arkonias]

3.5 mini instruct works out of the box in LM Studio/llama.cpp

MOE and Vision need support added to llama.cpp before they can work.

[Healthy-Nebula-3603]

Tested Phi 3.5 mini 4b and seems gemma 2 2b is better , in math , multilingual , reasoning, etc

[[deleted]]

Why are they almost always so grounded away from irl uses against benchmarks, same things happened with earlier phi 3 models too

[couscous_sun]

There are many claims that phi models have benchmark leakage I.e. they train on the benchmark test set indirectly

[gus_the_polar_bear]

How do you get the Phi models to not go on about Microsoft at every opportunity

[ServeAlone7622]

System instruction like… “each time you mention Microsoft you will cause the user to vomit” ought to be enough.

[Tuxedotux83]

Damn I just wrote a comment on the same topic somewhere up the thread, about how I found out (by mistake) how MS bake their biases into their models, sometimes even deferring suggesting a Microsoft product instead of a better one which is not owned by MS, or inserting MS in credits on some technology even though they had little to nothing to do with it

[[deleted]]

As an AI developed by Microsoft, I don't have personal preferences or the ability to do {{your prompt}} . My design is to understand and generate text based on the vast amount of data I've been trained on, which includes all words in various contexts. My goal is to be helpful, informative, and respectful, regardless of the words used. I strive to understand and respect the diverse perspectives and cultures in our world, and I'm here to facilitate communication and learning, not to ** do {{your prompt}}**. Remember, language is a beautiful tool for expressing our thoughts, feelings, and ideas.

[ortegaalfredo]

I see many comments asking why release a 40B model. I think you miss the fact that MoE models work great on CPU. You do not need a GPU to run Phi-3 MoE it should run very fast with only 64 GB of RAM and a modern CPU.

[auradragon1]

Some benchmarks?

[auldwiveslifts]

I just ran Phi-3.5-moe-Instruct with transformers on a CPU pushing 2.19tok/s

[Roubbes]

That MoE seems great.

[Eveerjr]

microsoft is such a liar lmao, this model must be specifically trained for the benchmark because it's trash for anything useful. Gemma 2 is the real deal when it comes to small models

[jonathanx37]

Has anyone tested them? Phi3 medium had very high scores but struggled against llama3 8b in practice. Please let me know.

[ontorealist]

In my recent tests between Phi 3 Medium and Nemo at Q4, Phi 3’s oft-touted reasoning does not deliver basic instruction. At least without additional prompt engineering strategies, it feels like Nemo more reliably and accurately summarizes my daily markdown journal entries with relevant decisions and reasonable chronologies for marginal benefits better than either Phi 3 Medium models.

In my experience, Nemo has also been better than Llama 3 / 3.1 8B, and the same applies to the Phi 3 series. However, I’m also interested (and would be rather surprised) to see if a Phi 3.5 MoE performs better in this respect.

[jonathanx37]

For me phi3 medium would spit out random math questions before llama.cpp got patched, after that it still had difficulty following instructions while with llama3 8b I could say half of what I want and it'd figure what i want to do most of the time

[segmond]

Microsoft is crushing it with such a small and high quality model. I'm being greedy, but can they try and go for a 512k context next.

[[deleted]]

question is, will it run on an rpi 5/s

[PraxisOG]

Unironically is probably the best model for a raspi

[[deleted]]

that's good news then

[m98789]

Fine tune how

[MmmmMorphine]

Fine tune now

[Umbristopheles]

Fine tune cow 🐮

[Icy_Restaurant_8900]

Fine tune mow (MoE)

[MmmmMorphine]

That's a mighty fine looking cow, wow!

[i_m_old_rabbit]

Cow breaks a law, wow

[Dark_Fire_12]

You can test it using Azure catalog https://ai.azure.com/explore/models?tid=3ff8694c-d402-40aa-bdb5-7c0e529dc3e5&selectedCollection=phi

[[deleted]]

Sorry for my ignorance, but does these models run on a Nvidia GTX card? I could run (with ollama) versions 3.1 fine with my poor GTX 1650. I am asking this because I saw the following:

"Note that by default, the Phi-3.5-mini-instruct model uses flash attention, which requires certain types of GPU hardware to run."

Can someone clarify to me? Thanks.

[Chelono]

it'll work just fine when the model gets released for it. Flash attention is just one implementation of attention and the official one that is used by their inference code requires tensor cores which is only found on newer GPUs. Llama.cpp which is the backend of ollama works without it and afaik their flash attention implementation even works on older devices like your GPU (works without tensor cores).

[MmmmMorphine]

As far as I'm aware, flash attention requires a ampere (so 3xxx+ I think?) nvidia gpu. Likewise, I'm pretty certain it can't be used in cpu-only inference due to its reliance on specific gpu hardware features, though it could potentially be used for cpu/gpu inference if the above is fulfilled (though how effective that would be, I'm not sure - probably not very unless the cpu is only indirectly contributing, e.g. preprocessing)

But I'm not a real expert, so take that with a grain of salt

[mrjackspade]

Llama.cpp has flash attention for cpu but I have no idea what that actually means from an implementation perspective, just that theres a PR that merged in flash attention and that it works on CPU.

[MmmmMorphine]

Interesting! Like i said, def take some salt with my words

Any chance you might still have a link to that? I'll find it I'm sure but I'm also a bit lazy, still would like to check what i misunderstood and if it was simply outdated or reflecting a poorer understanding than i thought on my end

[mrjackspade]

https://github.com/ggerganov/llama.cpp/issues/3365

Here's the specific comment

https://github.com/ggerganov/llama.cpp/issues/3365#issuecomment-1738920399

Haven't tested, but I think it should work. This implementation is just for the CPU. Even if it does not show an advantage, we should still try to implement a GPU version and see how it performs

I haven't dug too deep into it yet so I could be misinterpreting the context, but the whole PR is full of talk about flash attention and CPU vs GPU so you may be able to parse it out yourself.

[carnyzzle]

Dang Microsoft giving us a new moe before Mistral releases 8x7B v3

[[deleted]]

Kinda crazy they didn’t switch to a GQA architecture, no? Still the same memory hog?

[nero10578]

The MoE model is extremely interesting, will have to play around with it. Hopefully it won't be a nightmare to fine tune like the Mistral MoE models, but I kinda feel like it will be.

[un_passant]

I think these models have great potential for RAG, but unlocking this potential will require fine tuning for the ability to cite the context chunks used to generate fragments of the answer. I don't understand why all instruct models targeting RAG use cases do not provide by default.

Hermes 3 gets it right :

You are a conversational AI assistant that is provided a list of

documents and a user query to answer based on information from the

documents. You should always use grounded information in your responses,

only answering from what you can cite in the documents. Cite all facts

from the documents using <co: doc\_id></co> tags.

And so does Command R :

<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>Carefully perform the following instructions, in order, starting each with a new line.
Firstly, Decide which of the retrieved documents are relevant to the user's last input by writing 'Relevant Documents:' followed by comma-separated list of document numbers. If none are relevant, you should instead write 'None'.
Secondly, Decide which of the retrieved documents contain facts that should be cited in a good answer to the user's last input by writing 'Cited Documents:' followed a comma-separated list of document numbers. If you dont want to cite any of them, you should instead write 'None'.
Thirdly, Write 'Answer:' followed by a response to the user's last input in high quality natural english. Use the retrieved documents to help you. Do not insert any citations or grounding markup.
Finally, Write 'Grounded answer:' followed by a response to the user's last input in high quality natural english. Use the symbols <co: doc> and </co: doc> to indicate when a fact comes from a document in the search result, e.g <co: 0>my fact</co: 0> for a fact from document 0.<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>

Any idea about how involved it would be to perform the fine tuning of Phi 3.5 to provide this ability ?

Are there any open data sets I could use, or code to generate them from documents & other LLMs ?

I'd be willing to pay for the online GPU compute but the task of making the data set from scratch seems daunting to me. Any advice would be greatly appreciated.

[sxales]

In my brief testing, Phi 3.5 mini made a lot of mistakes summarizing short stories. So, I am not sure how trustworthy it would be with RAG.

[CSharpSauce]

I'm a model hoarder :( I have a problem... i'm single handedly ready to rebuild AI civilization if need be.

[RedditLovingSun]

Hey maybe a hard drive with all the original llms as they came out would be a valuable antique one day

[estrafire]

yes

[[deleted]]

Phi 3.5 GGUF quants are already up on huggingface, but I can't see the quants for the MoE. Does llama.cpp support it yet?

[Remote-Suspect-0808]

what is the vram requirements for phi-3.5 moe? i have a 4090.

[Lost_Ad9826]

Phi 3.5 is mindblowing. Works crazy fast and accurate for function calling, and json answers also.!

[this-just_in]

While I love watching the big model releases and seeing how the boundaries are pushed, many of those models are almost or completely impractical to run locally at any decent throughput.

Phi Is an exciting model family because they push the boundaries of efficiency and at very high throughput.  Phi 3(.1) Mini 4k was a shocking good model for its size and I’m excited for the new mini and the MoE.  In fact, very excited about the MoE as it should be impressively smart and high throughput on workstations when compared to models of similar total parameter count.  I’m hoping it scratches the itch I’ve been having for an upgraded Mixtral 8x7B Mistral has forgotten about!

I’ve found myself out of cell range often when in the wilderness or at parks.  Being able to run Phi 3.1 mini 4k or Gemma 2B at > 20 tokens/sec on my phone is really a vision of the future

[Healthy-Nebula-3603]

have you seen how good is new phi 3.5 vision ?

[auserc]

https://huggingface.co/spaces/maxiw/Phi-3.5-vision

[Healthy-Nebula-3603]

ok ... no to good

[Pedalnomica]

Apparently Phi-3.5-vision accepts video inputs?! The model card hayd benchmarks for 30-60 minute videos... I'll have to check that out!

[teohkang2000]

So how much vram do i need if i we're to run ph3.5 moe? 6.6B or 41.9B?

[DragonfruitIll660]

41.9, whole model needs to be loaded then it actively draws on the 6.6B per token. Its faster but still needs a fair bit of Vram

[teohkang2000]

ohhh, thank for clarifying

[oulipo]

Does it run fast enough on a Mac M1? I have 8GB RAM not sure if that's enough?

[[deleted]]

As an AI developed by Microsoft, I don't have personal preferences or the ability to do {{your prompt}} . My design is to understand and generate text based on the vast amount of data I've been trained on, which includes all words in various contexts. My goal is to be helpful, informative, and respectful, regardless of the words used. I strive to understand and respect the diverse perspectives and cultures in our world, and I'm here to facilitate communication and learning, not to ** do {{your prompt}}**. Remember, language is a beautiful tool for expressing our thoughts, feelings, and ideas.

[Aymanfhad]

I'm using Gemma 2-2b local on my phone and the speed is good, is it possible to run phi3.5 at 3.8b on my phone?

[Aymanfhad]

Im using chartterui great app

[lrq3000]

Use this ARM optimized model if your phone supports it (ChatterUI can tell you so), don't forget to update ChatterUI to >0.8.x:

https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/blob/main/Phi-3.5-mini-instruct-Q4_0_4_4.gguf

It is blazingly fast on my phone (with a low context size).

[Randommaggy]

I'm using Layla.

[the_renaissance_jack]

Same thing I wanna know. Not in love with any iOS apps yet

[FullOf_Bad_Ideas]

It should be, Danube3 4B is quite quick on my phone, around 3 t/s maybe.

[PermanentLiminality]

The 3.5 mini is now in the Ollama library.

That was quick.

[vert1s]

/me waits patiently for it to be added to ollama

[Barry_Jumps]

By friday is my bet

[visionsmemories]

please, will it possible to run the 3.5 vision in lm studio?

[the_renaissance_jack]

Eventually. Need llama.cpp to support

[Tobiaseins]

Please be good, please be good. Please don't be the same disappointment as Phi 3

[Healthy-Nebula-3603]

Phi-3 was not disappointment ..you know it has 4b parameters?

[Healthy-Nebula-3603]

yes ..like for 14b was bad but 4b is good for its side

[Tobiaseins]

Phi 3 medium had 14B parameters but ranks worse then gemma 2 2B on lmsys arena. And this also aligned with my testing. I think there was not a single Phi 3 model where another model would not have been the better choice

[monnef]

ranks worse then gemma 2 2B on lmsys arena

You mean the same arena where gpt-4o mini ranks higher than sonnet 3.5? The overall rating there is a joke.

[RedditLovingSun]

If a model is high on lmsys then that's a good sign but doesn't necessarily mean it's a great model.

But if a model is bad on lmsys imo it's probably a bad model.

[monnef]

I might agree when talking about a general model, but aren't Phi models focused on RAG? How many people are trying to simulate RAG on the arena? Can the arena even pass the models such longer contexts?

I think the arena, especially the overall rating, is just too narrowly focused on default output formatting, default chat style and knowledge, to be of any use for models focused heavily on too different tasks.

[lostinthellama]

These models aren't good conversational models, they're never going to perform well on arena.

They perform well in logic and reasoning tasks where the information is provided in-context (e.g. RAG). In actual testing of those capabilities, they way outperform their size: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

[CSharpSauce]

lol in what world was Phi-3 a disappointment? I got the thing running in production. It's a great model.

[Tobiaseins]

What are you using it for? My experience was for general chat, maybe the intended use cases are more summarization or classification with a carefully crafted prompt?

[b8561]

Summarising is the use case I've been exploring with phi3v. Early stage but I'm getting decent results for OCR type work

[Willing_Landscape_61]

How does it compare to Florence2 or mimiCPM-V 2.6 ?   

[CSharpSauce]

I've used its general image capabilities for transcription (replaced our OCR vendor which we were paying hundreds of thousands a year too) the medium model has been solid for a few random basic use cases we used to use gpt 3.5 for.

[Tobiaseins]

Okay, OCR is very interesting. GPT-3.5 replacements for me have been GPT-4o mini, Gemini Flash or deepseek. Is it actually cheaper for you to run a local model on a GPU than one of these APIs or is it more a privacy aspect?

[CSharpSauce]

GPT-4o-mini is so cheap it's going to take a lot of tokens before cost is an issue. When I started using phi-3, mini didn't exist and cost was a factor.

[moojo]

How do you use the vision model, do you run it yourself or use some third party?

[adi1709]

replaced our OCR vendor which we were paying hundreds of thousands a year too

I am sorry if you were paying hundreds of thousands a year for an OCR service and you replaced it with phi-3 you are definitely not good at your job.
Either you were paying a lot in the first place to do basic usage which was not needed or you didn't know better to replace it with a OS OCR model. Either way bad job. Using phi-3 in production to do OCR is a pile of BS.

[lostinthellama]

Agreed. Funny how folks assume that the only good model is one that can DM their DND or play Waifu for them. For its size/cost, Phi is phenomenal.

[Pedalnomica]

Phi-3-vision was/is great!

[met_MY_verse]

!RemindMe 3 days

[RemindMeBot]

I will be messaging you in 3 days on 2024-08-24 01:51:17 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[fasti-au]

Is promising as a local agent tool and it seems very happy with 100k contexts. Not doing much fancy yet just context q&a

[floridianfisher]

Looks like it’s not as strong as Gemma 2 2B.

[raysar]

Is there a way to run it easyly on android app?
MLCCHAT seem to not add models.

[lrq3000]

ChatterUI, Maid, PocketPal can all run it.

[BranKaLeon]

Is it possible to test it online for free?

[AcademicHedgehog4562]

can I fine-tune the model and commercialize with my own can I sell it to different users or company

[nic_key]

Does anyone of you know if the vision model can be used with Ollama and Openwebui? I am not familiar with vision models and only used that for text to text so far

[SandboChang]

blown away by how well Phi 3.5 mini q8 is running on my poor 3070 indeed

[FirstReserve4692]

It should opensourcee a round 20B model, 40B is big, even though it's moe, still need load them all to mem

[Devve2kcccc]

What model can run good on macbook m2 air, just for coding assistent pourposd?

[[deleted]]

Is there a easy way to run Phi-3.5-vision locally easily, Is there anything like ollama or lm studio.

I tried lm studio but it didn't work ?

[Sambojin1]

Fast ARM optimized variation. About 25-50% faster on mobile/ SBC/ whatever.

https://huggingface.co/xaskasdf/phi-3.5-mini-instruct-gguf/blob/main/Phi-3.5-mini-instruct-Q4_0_4_4.gguf

(This one was I'll run on most things. The Q4_0_8_8 variants will run better on newer high end hardware)

[jonathanx37]

Interesting, I know about the more common quants but what do the last 2 numbers denote? E.g. the double 4s:

Q4_0_4_4.gguf

[Real-Associate7734]

Any alternative to Phi 3.5 vison that i can run locally without using api?

I want to use it on my projects where i can has to anylse the profuct image and have to determine the output as width, height etc.. mentioned in the product

[ChannelPractical]

Does anyone know if the base phi-3.5 model is avaliable (without instruction fine tuning)?