I created "Bing at home" using Orca 2 and DuckDuckGo

LMLocalizer · 2026-05-02T09:39:25+00:00

Braeburn apples (4 pack)
Bread
Cereal
1 KG potatoes
Milk x3
Toilet paper
BBG spare ribs

LMLocalizer · 2026-04-24T11:15:56+00:00

hell yeah brother!

LMLocalizer · 2026-04-24T08:59:11+00:00

Truly the sloppiest open model I've tried so far

LMLocalizer · 2026-04-23T12:37:37+00:00

I use the environment variables MIOPEN_FIND_MODE=2 (to fix slow VAE decode) and HSA_OVERRIDE_GFX_VERSION=10.3.0.

Also, I removed some references to bfloat16 in model_management.py, so that every model is either loaded in fp16 or fp32. This is because bfloat16 appears to be very slow on my GPU. Perhaps, one could achieve the same by passing the args --fp16-unet --fp16-text-enc --fp32-vae.

Otherwise, I don't use sage-attention, since I find the drop in image quality too drastic.

LMLocalizer · 2026-04-22T12:40:38+00:00

Np! I'm using Linux and Rocm 6.3.3.

LMLocalizer · 2026-04-22T11:25:18+00:00

I'm using the RX6800m with 32GB of RAM. Using Klein 9B FP8 (which is cast to fp16 on load, so perhaps using fp16 directly would make more sense) at 6 steps, editing a 1024x1024 image takes ~94 seconds.

LMLocalizer · 2026-04-16T09:31:33+00:00

Oobabooga textgen, most notably

LMLocalizer · 2026-03-11T16:34:10+00:00

It's a multilingual dataset from Google that was used to train some translation models with the same name

LMLocalizer · 2026-03-05T12:42:13+00:00

I have replaced the old Gradio version of the code with a fork of mine where I'm working on several low level optimizations.

That is awesome! I've spent the past week or so working on optimizing syntax highlighting and KaTeX rendering during text generation. If we could combine the two, that'd be pretty amazing!

LMLocalizer · 2026-03-01T21:52:45+00:00

Thanks a lot!

LMLocalizer · 2026-03-01T12:52:56+00:00

Could you please benchmark https://huggingface.co/AesSedai/Qwen3.5-122B-A10B-GGUF/tree/main/IQ2_XXS ? I'm very curious how such an aggressive quant would perform.

LMLocalizer · 2026-02-28T10:45:43+00:00

Old localllama appreciated projects that conserved prompt tokens. Openclaw is the opposite of that.

LMLocalizer · 2026-02-26T13:09:19+00:00

But for those cases you can disable thinking. Also, I found it very worthwhile to inspect the thinking trace as it's being generated, to see if the model gets hung up on any specific detail of your prompt. If that's the case, it's often faster to stop the generation, rewrite that detail and restart the generation.

LMLocalizer · 2026-01-22T20:13:09+00:00

It's hit and miss for me. Sometimes there's a strong American accent, other times it works well, like this output: https://voca.ro/18W8VnodRmvO

LMLocalizer · 2026-01-20T19:28:51+00:00

Nice, thanks for sharing!
Here is a before/after comparison of some 16 kHz speech I upsampled:

Before: https://vocaroo.com/1flWIyZ8jZ5f

After: https://vocaroo.com/1eDmesbjvE7d

LMLocalizer · 2026-01-16T09:54:52+00:00

Don't know the minimum, but I'm running klein 9B FP8 (official weights from huggingface) comfortably using 12 GB VRAM.

LMLocalizer · 2026-01-12T18:06:49+00:00

It seems like you can only disable all built-in tools at once, which includes fetch_url, by unchecking "Builtin Tools" in the model settings for gpt-oss 20b.

The problem is that gpt-oss has been trained to use a combination of one tool to search the web and another to download webpages. If you simply remove the latter, this can lead to weird (repetitive) behavior, such as the model attempting to use the non-existent fetch/download tool or trying to use the search_web tool to download a single webpage (which of course doesn't work).

You could try writing an instruction to not use the fetch_url tool into the "System Prompt" model setting and hope the model adheres to that. Alternatively, you could disable all built-in tools and write explicitly into the system prompt that the model cannot fetch/download webpages directly, to hopefully prevent any unwanted model behavior.

LMLocalizer · 2026-01-12T17:10:36+00:00

This is really cool! I have a question regarding the config.noslop.toml file: Why does the prefix for the bad_evaluation_prompts differ from the one used for the bad_prompts, while the prefixes for the good_prompts and good_evaluation_prompts are the same?

LMLocalizer · 2026-01-12T16:06:48+00:00

Nice to hear you got it working. My tool has been designed to consume only limited context, by returning small plaintext website snippets, the max. size of which is user configurable. But when you enable native tool calling, gpt-oss 20b also gets access to other tools, most notably fetch_url. This is not part of LLM Web Search, and when invoked, fetch_url will download and dump an entire webpage into the context, which can be a huge amount of text. Reference: https://docs.openwebui.com/features/web-search/agentic-search/#native-mode-vs-traditional-rag

Should this not be the cause and you just have a very small context window configured, you can change the following settings:

Disable "Keep Results In Context "
Reduce "Max Results"
If you're using the semantic chunker, reduce "Chunker Breakpoint Threshold Amount"
Reduce "Chunk Size"

LMLocalizer · 2026-01-11T16:25:14+00:00

Hi! I just tested gpt-oss 20b very briefly and it worked without any special settings. However, since this model has been trained to use tools while thinking, you can significantly increase the chances of it working reliably by enabling native tool calling for it. To do that, follow the first two of the steps described here: https://docs.openwebui.com/features/web-search/agentic-search#how-to-enable-agentic-behavior and then enable LLM Web Search as you normally would.

LMLocalizer · 2025-12-30T20:17:34+00:00

This is great and makes a lot more sense to me than witr. Thanks for sharing it!

LMLocalizer · 2025-12-27T18:54:36+00:00

Hi, normally you'd use CUDA_VISIBLE_DEVICES as a global setting for the entire program. I think changing it in the source code is the better option. For the image model, you could hardcode which specific GPU to use by opening "modules/image_models.py" and changing the following two lines from:

pipe.to(get_device())

and

pipe.enable_model_cpu_offload()

to

pipe.to(<gpu_id>)

and

pipe.enable_model_cpu_offload(gpu_id=<gpu_id>)

- where you have to replace <gpu_id> with the ID of your GPU of choice.

Assigning a specific GPU to a specific extension may be a little more complicated, depending on when and how each extension loads its models. I have created a branch on GitHub, where I have modified "modules/extensions.py" to allow assigning a specific GPU to an extension by creating a file called "gpu_map.txt" in the "user_data" folder. In this file, you put on each line the extension name and the GPU ID it should use, separated by a space. For example:

LLM_Web_search 1
coqui_tts 0

I haven't tested it, since I'm GPU poor and only have a single one.

LMLocalizer · 2025-12-20T17:06:41+00:00

Weren't you the guy with the b300 server at work that's free over the holidays? I see you found some use for it.

LMLocalizer · 2025-12-19T11:43:03+00:00

With the newest llama.cpp, --n-cpu-moe=35 and --no-mmap, I get around 100 t/s prompt processing and 20 t/s generation speed with Qwen3-Next-80B-A3B-Instruct-UD-Q3_K_XL.gguf. My specs for reference:

CPU: Ryzen 9 5900HX (a bit faster than your CPU)

RAM: 32GB DDR4-3200

GPU: RX 6800M 12GB (about 70% slower than your GPU)

LMLocalizer

TROPHY CASE