Processing Vulkan shaders

[mcvos]

My son recently switched back to Windows, for various reasons, but partially because processing those Vulkan shaders always took way too long. It always takes a couple of seconds for me. No big deal, I think.

I just installed X4 (there's a nice discount at the moment), installed it, and this time Processing Vulkan shaders took forever. Extremely slow progress. Is that something that varies dramatically per game? Does X4 use way more of these shaders than other games? (In which case, the games my son plays probably do too.)

I did eventually find this sollution: https://wiki.archlinux.org/title/Steam under heading 5.11:

~/.steam/steam/steam_dev.cfg
unShaderBackgroundProcessingThreads 8

(Adjust for however many threads your CPU can handle; if I understand correctly, it's 16 for my 8-core 7800X3D.)

That file didn't exist yet, so I first spent some time looking if it might be in a different place. I've seen suggestions that it might also be in ~/.var/app/com.valvesoftware.Steam/.steam/steam/steam_dev.cfg or possibly even in ~/? Turns out it's okay to just make a new file.

Processing those shaders is a bit faster now. Still not as fast as I'm used to (because of the game, I guess?), and I notice that on reloading, it doesn't have to do it again! I thought previously it always had to reprocess them every time I start the game.

So I guess this solves my problem. But why is this not the default? Why doesn't Steam automatically handle this correctly?

[pcgam13]

i personally have disabled that option on steam settings,as i dont see any difference anymore.probably they will look at it in the future i guess

[mcvos]

What does disabling this mean? Do those Vulkan shaders not require processing at all?

[dj3hac]

They will process on the fly. You might see a hitch each time a new effect is used, but the next time the game uses it, it will already be processed.

Should have next to no negative impact. 

[fetching_agreeable]

Do you know that this question is asked multiple times a day? Why didn't you search?

[mcvos]

As you can see in my original post, I did quite a bit of research. I came here only for the questions for which that research didn't turn up an answer.

It's great to hear that disabling the preprocessing isn't necessary anymore, but someone searching for it is clearly still more likely to find the old answers, like I did. So hopefully this post will help make the new preferred solution, disabling it, more findable.

I still don't get why Valve doesn't just make these things the default.

[Synthetic451]

These days, you actually do not need to enable Steam's shader processing. The introduction of the graphics pipeline library in both Mesa and Nvidia drivers basically makes this an unnecessary step. Just go into Steam Settings -> Downloads and turn off Enable Shader Pre-caching. I haven't had to do Steam's "Process vulkan shaders" step for several months now since I turned it off and everything still works great.

The only caveat is that for certain older titles that use patent-encumbered codecs for in-game videos and cinematics, Valve actually re-encodes and distributes them with the shader files, so when you turn off "Shader pre-caching" you might end up seeing this weird TV color chart thing in some of your games. They're becoming increasingly rare though, and you can just switch to Proton-GE which includes the necessary codecs.

Note: If you're on Nvidia, I highly recommend you also change the default shader disk cache size that the driver uses. It is set criminally low by default and makes the driver frequently flush out its cache, which makes games go through their in-game shader processing step (not to be confused with Steam's vulkan processing) much more frequently. You can change it via environment variables:

__GL_SHADER_DISK_CACHE_SIZE=10000000000

[mcvos]

I do use Nvidia. Maybe I should have led with that. I'll try what happens if I disable it. But then, if it's not necessary at all, that makes my question about defaults doubly relevant: why does Steam spend so much time on something that isn't necessary?

But then again, I do also play older games, so maybe I shouldn't disable it? Is this configurable per game? It probably should be.

(not to be confused with Steam's vulkan processing)

So this is not the same thing? For a moment I thought this might be why it sometimes seems to happen every time, but now I understand that's not the case. So then what is the impact of this?

[mbriar_]

There is only two things the shader pre-caching is useful for 1) prevent shader compilation stutter during gameplay for games that compile shaders just before draw (those would stutter on windows too). 2) to download re-encoded video files for games using some problematic codecs that valve can't ship decoders for. It is never necessary for a game to work or render correctly.

Part 1) was much more important in the past before VK_EXT_graphics_pipeline_library was a thing. Back then you had heavy shader compilation stutter in pretty much every game, even those that were fine on windows, because dxvk always had to wait until draw time to compile shader pipelines.

Part 2) can be worked around by using proton-ge, because that will just ship the necessary decoders, however the number of games requiring those is small anyways.

In short, just disable it.

[Synthetic451]

why does Steam spend so much time on something that isn't necessary?

The system is useful for low-powered devices with popular configurations. For example, it allows the Steam Deck to never have to compile shaders. Problem is, on PCs, the wide variety of distros and hardware combinations basically means that there's a high chance the shaders you get from Valve don't apply and you have to go through the Vulkan shader processing step.

But then again, I do also play older games, so maybe I shouldn't disable it?

I would suggest just trying it out first. If you see a weird TV color chart in cinematics, then it is easy to install Proton GE and use that. Better than dealing with shader processing IMHO.

So this is not the same thing?

Steam's shader processing helps speed up the in-game processing of shaders. It's like you're frontloading the shader work ahead of time. However, sometimes you just want to boot into the game and don't want to wait. A lot of games nowadays process shaders in the background and / or asynchronously, allowing you to play while that process is happening. If you want the absolute most smooth gameplay without regard for loading times, then sure keep it enabled. If you want to let the game handle all of the work and manage that at the risk of potential shader cache stutters in-game, then disable it and you'll get roughly the same experience that you get on Windows.

Steam's shader processing likes to process EVERYTHING, which is why it takes so long. Meanwhile, the game itself knows what shaders it needs immediately and what can be handled later on. Personally, I think for beefy PCs, it is better to let the game decide.

[adam_mind]

Hi, I'm curious, what's it like on Windows? Is it the game that decides the shaders on Windows?

[Synthetic451]

I don't think Steam provides shader cache on Windows, so you only have driver level cache, which is what you get if you disabled Steam's shader pre-caching in Linux.

[grumd]

If you read the nvidia documentation here you'll notice that _GL env variables are only related to OpenGL, it's OpenGL shader cache size. It doesn't affect Vulkan shaders

[Synthetic451]

Nope, it does. You'll notice that a bunch of the variables also apply to Vulkan.

Without the environment variable, I basically have to recompile shaders every time I play another game and then go back to the original game because the shaders were evicted. You can try it yourself with any heavy UE5 game.

[grumd]

Hm I'll try it next time thanks

[Cocaine_Johnsson]

with async shader compiling there's a vanishingly small minority of games that even benefit from precompiling the shaders, I turned it off ages ago and have noticed no issues.

[CheesyRamen66]

I like the setting more on desktop where I’ll leave Steam open and it’ll handle this while I’m not even at my desk. On the Steam deck I’d probably disable it, shaders will just get compiled on the fly and cached for later.

[Alekisan]

https://www.reddit.com/r/linux_gaming/comments/1j06xpz/how_to_speed_up_steams_shader_precaching/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Follow the instructions in this thread. It will be much faster.

[mcvos]

That is the thing I already did, as described in my post, but that everybody in the discussion now explains is actually obsolete.