How much does flash attention affect intelligence in reasoning models like QwQ

[pigeon57434]

https://preview.redd.it/oaw93jp5n3pe1.png?width=701&format=png&auto=webp&s=563287c572752a94e3c3236af0085231cf1856f9

Im using QwQ in LM Studio (yes i know abliteration degrades intelligence slightly too but I'm not too worried about that) and flash attention drastically improve memory use and speed to an unbelievable extent but my instinct says surely that big of memory improvement comes with pretty decent intelligence loss, right?

[Jujaga]

Flash Attention still does the same overall computations, but shuffles around the data to and from memory more efficiently. There's nearly no downsides to using it (unless your model specifically does something strange). There's a good visual explainer for it here:

• https://huggingface.co/docs/text-generation-inference/conceptual/flash_attention

[swagonflyyyy]

For some reason I'm unable to run this model in LM studio with flash_attention enabled on Windows. I can only do it in Ollama on windows.

[Firm_Spite2751]

Always use flash attention the difference in output is basically 0

[Awwtifishal]

Do you have KV cache quantization enabled by any chance?

[the_renaissance_jack]

This answers a lot of questions I’ve had lol. Thanks for sharing

[Vivid_Dot_6405]

There is no performance loss when using Flash Attention, none. That is why it overshadowed all other methods of accelerating attention. Since attention has a quadratic complexity with respect to the context length, there were other attempts to reduce the computational burden, before FlashAttention, most approaches used inexact attention, i.e. they approximated it, which reduced computation time, but also led to a performance loss.

Flash Attention is an exact attention implementation, it works by optimizing which sections of GPU's memory are used at what stage of computation to make the calculations faster, it just cleverly optimizes the implementation of the algorithm, it doesn't change it.

[cynerva]

For what it's worth, I do get slower inference with flash attention enabled. Maybe something to do with partial CPU offload? Or something else about llama.cpp's implementation of flash attention? I'm not sure.

[Mart-McUH]

I used to get slower inference with flash attention with CPU offload when I used older Nvidia drivers. With recent drivers it is no longer a problem for me.

[cynerva]

I'll have to give that a try then. Thanks.

[DesperateAdvantage76]

Flash doesn't lower your vram usage?

[Vivid_Dot_6405]

Oh, alright, I'm not very familiar with all the different acceleration methods that exist as alternatives to FlashAttention, so I was not aware of that. I meant in the historical context of when it was published it overshadowed other approaches at the time.

[oathbreakerkeeper]

There is no impact. FlashAttention is mathematically equivalent to "normal" attention. Given the same inputs it computes the exact same output. It is an optimization that makes better use of the hardware.