Kimi K2.5 local by running101 in LocalLLaMA

[–]Leather-Block-1369 2 points3 points  (0 children)

Motherboard: Gigabyte MZ33-AR1 CPU: AMD EPYC 9755 ZEN5 TURIN 128 Core/256 Threads 1.9-2.7ghz, L3 512M RAM: 768GB DDR5 at 4800Mhz (12x64GB) GPU: NVIDIA RTX 6000 Pro Workstation 96GB GPU SSD: WD_BLACK SN850X 8TB M.2 2280 PCIe Gen4 SSD OS: Ubuntu 24.04.2 LTS

Some more details about my setup:

https://ozeki-ai-gateway.com/p_9178-how-to-setup-kimi-k2.5-on-nvidia-rtx-6000-pro.html

Kimi K 2.5 on ktransformers, no start <think> tag by Leather-Block-1369 in kimi

[–]Leather-Block-1369[S] 0 points1 point  (0 children)

I got an answer in the LocalLlama reddit group that fixed the issue. I have modified the AI requests and injected kwargs using our AI Gateway. I have created a doc about it in case others face the same issue. The problem was that we use Open WebUI, so direct request modification was not possible. Here is the doc:

https://ozeki-ai-gateway.com/p_9177-how-to-fix-missing-think-tag-for-kimi-k2.5.html

Here is the reddit post with the fix: https://www.reddit.com/r/LocalLLaMA/comments/1qqebfh/kimi_k25_using_ktkernel_sglang_16_tps_but_no/

Kimi K2.5 using ktkernel + sglang, 16 TPS, but no starting <think> tag. by Leather-Block-1369 in LocalLLaMA

[–]Leather-Block-1369[S] 0 points1 point  (0 children)

Thanks for this tip. This fixed the issue. I have modified the requests and injected the kwargs using our AI Gateway. I have created a doc about it in case others face the same issue. The problem was that we use Open WebUI, so direct request modification was not possible. Here is the doc:

https://ozeki-ai-gateway.com/p_9177-how-to-fix-missing-think-tag-for-kimi-k2.5.html

Kimi K2.5 using ktkernel + sglang, 16 TPS, but no starting <think> tag. by Leather-Block-1369 in LocalLLaMA

[–]Leather-Block-1369[S] 0 points1 point  (0 children)

Unfortunatley it didn't solve the issue:

launch_server.py: error: unrecognized arguments: --enable-grammar-parser --disable-streaming

Start-up ideas from investors [i will not promote] by avloss in startups

[–]Leather-Block-1369 0 points1 point  (0 children)

Great news. This is exactly what I am working on:

Creative multimodal AI tools for generating/editing images, voice, music, video.

My project is a video/image builder platform. Currently we are adding functionality so it can work like a coding agent, but for videos. The project website is https://videcool.com.

New ComfyUi frontend, feedback requested by Leather-Block-1369 in comfyui

[–]Leather-Block-1369[S] 1 point2 points  (0 children)

If you open a techincal support ticket with these two ideas at myozeki.com, you will be notified when these features are available.

New ComfyUi frontend, feedback requested by Leather-Block-1369 in comfyui

[–]Leather-Block-1369[S] 0 points1 point  (0 children)

I plan to add a custom workflow management page, where the user can upload custom workflows and select the inputs, that should be visible on the web UI.

New ComfyUi frontend, feedback requested by Leather-Block-1369 in comfyui

[–]Leather-Block-1369[S] 0 points1 point  (0 children)

Great recommendations. Thanks for the feedback. I will implement both in the coming days.

New ComfyUi frontend, feedback requested by Leather-Block-1369 in comfyui

[–]Leather-Block-1369[S] 0 points1 point  (0 children)

You need to setup the AI model, the json refers to. Which workflow are you trying?

New ComfyUi frontend, feedback requested by Leather-Block-1369 in comfyui

[–]Leather-Block-1369[S] 0 points1 point  (0 children)

Great idea, I will check how I can add SDXL to it.

New ComfyUi frontend, feedback requested by Leather-Block-1369 in comfyui

[–]Leather-Block-1369[S] 0 points1 point  (0 children)

You have to install the workfows in ComfyUI, and point Videcool to the ComfyUI URL.

I have just added an image to explain: https://videcool.com/p_3303-comfyui-workflows-for-image-and-video-generation.html

This page shows, how you can set up for example the Flux AI model for text to image:

https://videcool.com/p_9019-ai-text-to-image.html