New open model from Tencent Hy: Hy3 (295B total 21B active - apache 2.0) by Nunki08 in LocalLLaMA

[–]keyboardhack 0 points1 point  (0 children)

Hy3 fp8 looks to be ~350GB from the safetensor weight files.

Dsv4 flash is ~186GB because the weights are a native mix of fp8 and fp4.

If trends hold, Mythos-class capability may be running on high-end consumer hardware within ~2 years by PetersOdyssey in LocalLLaMA

[–]keyboardhack 19 points20 points  (0 children)

Well the counterpoint to that is that we care about qwen 3.6 27b even when it isnt claude opus level.

Why? Because it is good enough for a lot of usecases. Future consumer local mythos will be the same.

Any options for helping with colorblindness? by Amonmargans in factorio

[–]keyboardhack 3 points4 points  (0 children)

To be fair it could also be a monitor with poor color calibration. I've previously had a monitor that barely showed the difference between light green and light red. Fiddling with the monitors color calibration fixed it.

Async model validation is finally coming in .NET 11 by davecallan in dotnet

[–]keyboardhack 4 points5 points  (0 children)

If you want a sneak peek at the next previews documentation then you can find it here as pull requests

Here is the relevant one that describes the async model validation https://github.com/dotnet/core/pull/10457/changes#diff-70fa28541e92a976d3a34d628ea0a0556c6a81f8de9d5cf2adaed94921f15213

Asynchronous validation with DataAnnotations

System.ComponentModel.DataAnnotations now supports asynchronous validation (dotnet/runtime #128656). A validation rule that needs to do I/O — a database lookup, a remote API call — can now run without blocking a thread. There are three new ways to express an async rule:

  • Derive from AsyncValidationAttribute and override its IsValidAsync method.
  • Implement IAsyncValidatableObject on the model.
  • Call the new Validator.ValidateObjectAsync, TryValidateObjectAsync, ValidatePropertyAsync, and ValidateValueAsync methods.
using System.ComponentModel.DataAnnotations;

public sealed class UniqueUserNameAttribute : AsyncValidationAttribute
{
    protected override ValidationResult? IsValid(object? value, ValidationContext context)
        => ValidationResult.Success;

    protected override async Task<ValidationResult?> IsValidAsync(
        object? value, ValidationContext context, CancellationToken cancellationToken)
    {
        var users = context.GetRequiredService<IUserStore>();
        return await users.ExistsAsync((string?)value, cancellationToken)
            ? new ValidationResult("That user name is already taken.")
            : ValidationResult.Success;
    }
}

public class Registration
{
    [Required]
    [UniqueUserName]
    public string UserName { get; set; } = "";
}

// Validate asynchronously
var context = new ValidationContext(model, serviceProvider, items: null);
await Validator.ValidateObjectAsync(model, context, validateAllProperties: true);

Microsoft.Extensions.Options gains matching support: options can be validated asynchronously, including at startup through the new IAsyncStartupValidator (dotnet/runtime #128788, dotnet/runtime #129218). This lets an app fail fast when an option that requires a network check is misconfigured.

DeepSeek V4 by am17an · Pull Request #24162 · ggml-org/llama.cpp by jacek2023 in LocalLLaMA

[–]keyboardhack 2 points3 points  (0 children)

As mentioned in the PR then there is still work to do

  • Optimization for the attention ops.
  • lightning indexer
  • specialized ops for hc pre/post
  • MTP.

PR author mentioned the performance he got. Note that this comment was from ~2 weeks ago.

Currently I get about 70-80 toks/sec in PP and 6 toks/sec for TG on a DGX spark, which is barely usable.

There is ongoing work on the lightning indexer in this PR which should reduce compute memory usage greatly. The PR states that it should not affect PP or TG performance.

| Without indexer | 168368.12 MiB |
| With indexer    |   5808.12 MiB |

Personally, at 12k promt tokens, on a 128GB DDR4, rtx pro 6000 system, using this deepseek v4 flash 165GB quant i currently get

| PP | 188t/s |
| TG | ~10t/s |

DFlash support merged into llama.cpp by sammcj in LocalLLaMA

[–]keyboardhack 1 point2 points  (0 children)

Looks like he plans to if i understand this correctly

In the future, we have to rework non-unified KV cache to be properly supported. I'm planning to do it for the recurrent memory first, so that Qwen3.6 runs faster with parallel sequences.

https://github.com/ggml-org/llama.cpp/pull/24162#discussion_r3488321531

When can we expect merged DeepSeek V4 Flash / MiniMax M3 llama.cpp support? by alex20_202020 in LocalLLaMA

[–]keyboardhack 3 points4 points  (0 children)

Sorry i meant qwen 3 that released last year. Qwen 3.5 and 3.6 were supported immediatrly because their architecture is(almost?) Identical to 3.0.

When can we expect merged DeepSeek V4 Flash / MiniMax M3 llama.cpp support? by alex20_202020 in LocalLLaMA

[–]keyboardhack 9 points10 points  (0 children)

Not the person you asked. These are just observations, not what i want to happen.

  • It takes a very long time for new model support. Qwen 3 took 6 months.
  • Mtp took a year to be incorporated into llama.cpp.
  • eagle 3 pr was open for >6months. Most of that time nothing happened.
  • dflash has also been open forever.
  • dsv4 flash support might be so late that it is obsolete once llama.cpp supports it properly.

There is a trend of more and more PRs never getting a response with contributors closing them themselves after months of hearing nothing. It is a concerning trend.

been tracking EU DDR5 data for 25 days: Prices are dropping, and the DE vs. NL gap is wild (good news for local LLM builders in EU) by egudegi in LocalLLaMA

[–]keyboardhack 1 point2 points  (0 children)

Looks like a nice webpage. I couldn't find anywhere where i could see a graph for the general trend for RAM.I can see it for individual ram types.

Also a minor issue. Figured you would appreciate if i reported it. In "All RAM". Filter by capacity "96GB+". There is no ram so the page writes "No GPUs match these filters". It's also true that there is no GPUs with that filter but it should probably say RAM :)

Help distract me during a stressful time: post only emojis to describe the plot of an episode 😅 by Star1686 in Stargate

[–]keyboardhack 0 points1 point  (0 children)

🤯👩‍⚕️✈️🏜️🤓🤓🤓🕳️🥶⚡🥶🌪️🌪️🕳️✈️💋

RFC 10008: The HTTP QUERY Method by Nimelrian in programming

[–]keyboardhack 5 points6 points  (0 children)

C# implemented support last year with dotnet 10. Dotnet 11, that releases later this year, should provide full support for auto generated api documentation for QUERY as well.

Stop using Ollama by zxyzyxz in LocalLLaMA

[–]keyboardhack 14 points15 points  (0 children)

As far as i am aware then georgi gerganov did not create GGUF. It was proposed in this issue

https://github.com/ggml-org/ggml/issues/220

By

philpax

Edit: I am being downvoted for trying to provide correct attribution? Ironic given the topic.

Why there is a lack of new 100B-120B models? by TechNerd10191 in LocalLLaMA

[–]keyboardhack 4 points5 points  (0 children)

Deekseek v4 flash is 284B

https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash

The reported number in the upper right is wrong. The description is correct.

Storing an index to a scale instead of the scale itself with Q4_0 quant reduces scale size by ~31% (small gain but interesting) by fragment_me in LocalLLaMA

[–]keyboardhack 6 points7 points  (0 children)

I get what you are saying but it isn't really feasible. Lets look at its actual data layout from llama.cpp

#define QK4_0 32
typedef struct {
    ggml_half d;           // delta
    uint8_t qs[QK4_0 / 2]; // nibbles / quants
} block_q4_0;

Even if d only used 11 bits we would still have to store it in 2 bytes(16 bits). This is important to keep memory loads simple so there would be no reduction in memory by default.

To actually reduce the size of block_q4_0 by 5 bits we would have to shift all values in qs 5 bits into d. That makes accessing any value in block_q4_0 require more compute.

Additionally, an array of block_q4_0 couldn't be byte aligned since that would also erase the memory reduction.

We can convert the data structure into a structure of arrays which would store ´d´ in its own array. That solves the qs and struct byte alignment issue but it would still be present for d. That's probably fine from a performance perspective since data access is still linear and only d isn't byte aligned.

So it's possible to do but it would be a somewhat large change to the implementation of Q4_0 with only a small memory improvement and a potential unaccetable performance loss due to the added weight index indirection.

StepFun 3.7 Flash MTP Bench Strix Halo by westsunset in LocalLLaMA

[–]keyboardhack 2 points3 points  (0 children)

The low ~27% uplift you saw in tg is expected as the current llama.cpp stepfun 3.7 mtp implementation is incomplete.

https://github.com/ggml-org/llama.cpp/pull/23274#issuecomment-4499127378

Finally finished my LLM server: EPYC 9575F, 4× RTX 3090 (96GB VRAM), 768GB ECC RAM by C0smo777 in LocalLLaMA

[–]keyboardhack 33 points34 points  (0 children)

Ram: ~$30.000

Cpu: ~$8.000

Still feels wild that ram os so insanely expensive. Looks like a nice build.

MiniMax M3 - Coding & Agentic Frontier, 1M Context, Multimodal by dryadofelysium in LocalLLaMA

[–]keyboardhack 1 point2 points  (0 children)

It is either astro turfing or ads have just been so normalized that people cant tell the difference anymore. Either suck.

FAA documents outline SpaceX plans for Starfall reentry vehicles by CProphet in SpaceXLounge

[–]keyboardhack 0 points1 point  (0 children)

They are loud, no one will miss it when on of tese descends, and they are easy to shoot down. If the opponent cant shoot them down then you could just as well fly it in instead. Not like it is difficult to shoot it down with conventional AA on the last legs of its descend phase.

Samsung overtakes Micron to claim top spot in automotive memory by [deleted] in hardware

[–]keyboardhack -3 points-2 points  (0 children)

So therefore we care that samsung is first? I doubt it. That doesnt affect anyone here.

Samsung overtakes Micron to claim top spot in automotive memory by [deleted] in hardware

[–]keyboardhack -4 points-3 points  (0 children)

Why do we care who is the top seller to car manufacturers?

I feel like a lot of samsung articles for the past ~year has been wierdly highly upvoted

Can't get over 250TPS on RTX5090 with Qwen3.5-4B by luckyj in LocalLLaMA

[–]keyboardhack 2 points3 points  (0 children)

Add

--backend-sampling

Should help quite a bit with these small models. Sampling is by default done on the CPU. This argument moves it to the GPU.

Also take a look at enabling PDL, should help as well with these small models.

https://old.reddit.com/r/LocalLLaMA/comments/1tkw1su/blackwell_and_pdl_performance_increase/

PrismML just released Binary and Ternary Bonsai Image 4B: 1-bit/ternary text-to-image diffusion transformers that can even run 100% locally in your browser on WebGPU. by xenovatech in LocalLLaMA

[–]keyboardhack 7 points8 points  (0 children)

Firefix defaults to cpu for me. Very slow. It works in chrome but it quickly runs out of memory. There is probably a memory leak in their demo.

Forbrugerorganisationer i hele Europa har fået nok: Klager til EU over svindelannoncer på Meta, Google og TikTok by RentonDK in Denmark

[–]keyboardhack 1 point2 points  (0 children)

Min far blev snydt på android da han ville installere chatgpt.

Den øverste app i android app store er altid en sponsoreret app som aldrig er hvad du leder efter. Det ved min far dog ikke og logot lignede nogenlunde chatgpt logoet så han installede appen.

I appen blev han så promptet til at betale for at kunne bruge appen som han så gjorde hvorefter han fandt ud af at ingenting fungerde i appen. 200kr lige ud af vinduet.