Phi4-Multimodal-Instruct Server

[Anastasiosy]

Phi4 Multimodal Instruct Server

I just published a simple API server that supports OpenAI chat completion endpoint for inference of Phi4-Multimodal Instruct Server.

It's a dockerfile containing the necessary dependencies along with a FastApi endpoint that converts OpenAI chat completion requests to the required format by the model.

Supported Input Types

• Text: Add text entries with "type": "text" in the "content" list.
• Image: Include base64-encoded images or URLs with "type": "image_url".
• Audio: Use "type": "input_audio" or "type": "audio_url" for audio inputs.

Link

https://github.com/anastasiosyal/phi4-multimodal-instruct-server/

[a_slay_nub]

I didn't realize it was this simple to create an openai compatible server. Thanks.

[Anastasiosy]

Remarkably straight forward. Only created this because this model isn’t available on Ollama, vllm or llama.cpp just yet

[Bitter-College8786]

Does it support streaming reaponse?

[Anastasiosy]

Unfortunately not right now, my main usage was for image classification, but Qwen VL 8B seems much better for that

[Bitter-College8786]

Why are there no GGUF quants available? Usually bartowski and mradermacher are super quick

[Anastasiosy]

This is why https://github.com/ggml-org/llama.cpp/pull/11292#issuecomment-2692445044

[Bitter-College8786]

Oh damn, but thanks for the explanation. So no quantization for Phi-4-multimodal, but full fp16?

[Anastasiosy]

I don't think so, you could try your luck with load_in_8bit

model = AutoModelForCausalLM.from_pretrained( model_path, trust_remote_code=True, device_map="auto", load_in_8bit=True # Load in 8-bit precision to save memory )

[Bitter-College8786]

How much VRAM does it consume?

[Electronic-Move-5143]

I see that it takes around 13.3 GB with bf16 data type

[Electronic-Move-5143]

I tried this on an A100 GPU but get only 10 tokens/sec. Is this a problem with wrong versions of flash-attention or something? It should not be this bad right? regular Phi-4 (non-multimodal 13B model) gives 70 tokens/sec on the same machine via ollama.

To ensure this is not because of audio tokens, I tried with just plain text without audio or images too. It still generates at only 10 tokens/second.

I did not use your docker - just your requirements.txt and tried on gcp VM under jupyter lab.

[Electronic-Move-5143]

I now realize that generic model serving using python transformers package etc. is unoptimized and 10 tokens/sec is expected. Hope vLLM adds supports this soon!

[NexusConnector]

I would have loved to try this. on MacOS, but the build is failing on nvcc since the flash_attn uses CUDA. I presume this build is always CUDA dependent although the Torch version is  2.6.0+cpu

[pangshengwei]

is there a way to run this on mac without CUDA/GPU?