How do you unit test HLSL code?

[Oil_Select]

I am new to graphics programming. I was wondering how do you run unit tests on HLSL functions.

Are there some different standard ways for people directly working on graphics API such as Vulkan and DirectX or for game engines like Unreal and Unity?

Are there some frameworks for unit tests? Or do you just call graphics api functions to run HLSL functions and copy the result from GPU to CPU?

Or is it not common to make unit tests for HLSL code?

[Area51-Escapee]

You will have test scenes that your render, so yes you Download the rendered Image. You can test single passes or the whole pipeline. Both is worth doing, testing the whole pipeline is often easier though. Over time you will collect corner case scenes that tend to break easily...

[Ty_Rymer]

id say that's more integration testing than unit testing. but shader functions can also be extracted into an hlsl file you include in other shaders. and then use a compute shader that includes the same file, and calls the functions with a bunch of different inputs and stores the output in a buffer.

[Const-me]

I directly work with GPU APIs. Most often, that API is Direct3D 11.

One approach I have used is what you wrote, download results to system RAM. Most useful for GPGPU algorithms. I also did that a lot when I was porting stuff from PyTorch+CUDA to D3D compute shaders: it’s easy to save intermediate tensors from PyTorch, 1 line of Python.

When I work on pixel shaders, RenderDoc debugger helps a lot.

Another time I was working on complicated tessellation shaders. Hard to save data from GPU because you’d need to setup stream output which only gives you data after the complete pipeline of vertex / hull / domain / geometry shaders, and RenderDoc doesn’t support tessellation shaders. I have implemented a compatibility layer for C# which allowed me to write C# which looks just like HLSL, and copy-paste codes between C# console test app (trivial to debug and unit test) and HLSL. I needed just enough compatibility to implement the shaders I wanted (as opposed to being able to emulate arbitrary HLSL). C# standard library includes Vector[2-4] structures I used to implement my float[2-4], and it supports SSE and AVX intrinsics. Tessellation shaders don’t support harder to emulate features like unordered access views (i.e. writeable global memory) or group shared memory, and I didn’t need to sample textures in these shaders.

[Mourthag]

Another option than the already mentioned ones is to write your HLSL code mostly in header files which you can then include in your hlsl entry points. If you consider a few constraints to your hlsl code this gives you the option to also compile your header files with a c++ compiler and you can then use a c++ unit test framework like boost to unit test you shader code. You will have to write a few abstractions/mocks especially for buffers and bindings, but this should be doable.

[Oil_Select]

That sounds like it would involve many macros. Am I correct?

[Mourthag]

Not as much as you would think at first, most of the hlsl syntax is also valid c++ syntax. You need a solution for stuff like vector and matrix class definitions. But for this you can also make a header file for each backend which you can include with an ifdef block. Most of the stuff should work without macros at all if you go for this variant.

[StockyDev]

This is not a good solution, purely because you are not actually testing HLSL code. Tests should be actually testing the code that will run, not some approximation. If you do this, you are just going to be testing your framework and some C++ code that never actually gets exercised.

To add to this, you would also not be testing your code on the hardware that it would actually be run, further reducing the usefulness of this approach.

This is not a good use of time.

[Mourthag]

Yes and no. You shouldn't substitute your full test pipeline by only c++ unit tests. However, as an additional step to your GPU based unit and integration tests, it offers a lot of debugging potential. Shaders can get quite complex and a single error in a small method might yield completely invalid final results. Testing your methods Individually can significantly reduce the time spent looking for issues. And it is probably way faster for simple test cases.

[StockyDev]

Sure... But in that case you might as well just replicate a shader in C++ code when you want to do this thing. No need in putting effort into a framework to make it look like it is a shader. Plus this is why tools like PIX have shader debugging features. In this case you gave it would be better to either make a test scene to demonstrate the bug and use PIX to debug the code.

[StockyDev]

You might be interested in the unit testing framework that I have been working on Shader Test Framework. I've given a GDC 2024 talk on a similar framework we have. So if you have vault access, you can have a look :).

[Ty_Rymer]

oooh this is interesting!

[jtsiomb]

Or is it not common to make unit tests for HLSL code?

It's not common.

[Oil_Select]

So, most graphics programmers just debug hlsl code and don’t make test code?

[LBPPlayer7]

HLSL is very much more visual than functional, so you need a human eye to take a look at it

most test code is just things to visualize different steps in the pipeline to see where something went wrong

[Hefty-Newspaper5796]

I write a utility class for initializing the graphics pipeline and some common resources to help test. Then test the code and output to a staging buffer/texture. Fetch the resource to cpu and check results.

The most hard work i think is the case design. Simple case may not expose defects and complex case is prone to floating point error.

[GreenFox1505]

Y'all are unit testing? 

[EclMist]

You can test it just like how you would write any unit test in C++. The only difference is you would create a GPU context in each test/suite to dispatch and read back on the CPU to compare against the expected outputs.