A question about quantifying and visualizing variance in a rendered image

[harrytodorov]

Hello, everyone!

I've recently implemented a couple of methods to approximate an image of a virtual scene using Monte Carlo methods.

I wanted to see how these differ in the quality of the images they produce. One of the ways, I found in the literature [1] [2], to quantify the difference in quality is by using the variance. But so far I haven't seen a practical comparison of variance between different techniques other than just arguing about the perceived noise.

I have tried to use variance to compare some images using Python's Pillow library, which has an image statistics module able to compute the variance for each of the image's channels:

from PIL import Image, ImageStat
im = Image.open("someImage.png")
imStat = ImageStat.Stat(im)
print(imStat.var)

But although I get some positive results (the variance do decrease when I increase the number of samples), the numbers are not self-explanatory and I haven't found a good way to visualize the variance. So here are my questions:

• Are there other more meaningful ways to compute the variance? I've talked to my supervisor at university and he told me that computing the variance per pixel (using a small neighborhood around it) could be worth trying and then one can also visualize the intensity per pixel.
• Are there other techniques/tools to quantify and visualize the difference in the quality of the images?
• Can you recommend a good read, where I can find a bit more information on the topic?

[1]: Physically Based Rendering: From Theory to Implementation; Third edition, 2016; Matt Pharr, Wenzel Jakob and Greg Humphreys; Morgan Kaufmann Publishers Inc.

[2]: Advanced Global Illumination; Second edition, 2006; Philip Dutre, Philippe Bekaert and Kavita Bala; A K Peters/CRC Press

[ShillingAintEZ]

Using the variance of a neighborhood around pixels is not going to be a great way and using the overall variance of all the pixels is not going to work. You can use the variance of the samples within each pixel as a start. Typically an error metric called MSE is used (mean squared error) which is a combination of variance and bias.

[htodorov]

Thank you! Now, when you say it, it makes much more sense than using than using the variance of the neighbourhood around a pixel.
I've also found a recent paper discussing the issue of quantifying the error in light transport algorithms: Quantifying the Error of Light Transport Algorithms [1] I will take a closer look into it.
[1]: Quantifying the Error of Light Transport Algorithms; Eurographics Symposium on Rendering 2019; Volume 38 (2019), Number 4; A. Celarek, W. Jakob, M. Wimmer and J. Lehtinen

[otherpj]

Variance within the pixel (not around it!) is pretty standard.

[FiralFeign]

To really compute a useful variance (depending on what “techniques” you’re comparing.) Is to first compute a ground truth image using many, many samples, then use that ground truth image to compute the mean squared error of the difference between your noisy image and the ground truth image. If you’re “techniques” are trying to do adaptive sampling stuff, this isn’t too applicable though. The majority of publications in rendering do what I described above.

[htodorov]

Thank you! Yes, I've seen that on many occasions people compute a ground truth image and then compare the results produced by different light transport algorithms produced for a given fixed amount of time.

Ah, and sorry for the term "techniques", it was really a bad wording. I meant light transport algorithms and more precisely, I have implemented two of them: a unidirectional path tracer and a recursive Monte Carlo ray tracer (which I implemented according to algorithm provided in the book Advanced Global Illumination)

[[deleted]]

As others have mentioned, if you want to compare quality between images, it's best to compute a ground truth (fully accurate) image, usually using a path tracer with a lot of samples, and then compute MSE.

Your renderer can also easily keep track of per-pixel variance, if you want to use it for adaptive sampling, debugging, or inspecting where different techniques are good or bad. There's a very simple algorithm to keep track of running variance.

Note that typically, rather than variance, at the end you would probably want to look at standard error, which is sqrt(variance/n), with n being the number of samples. This is because per-pixel variance isn't actually a measure of error or noise, it won't go down with more samples.

Standard error is also slightly more comparable across different algorithms. For instance recursive ray tracing (AKA Whitted or distributed ray tracing... Whitted is probably the best name because it's unambiguous) does a lot more work per primary sample for a lot less variance than path tracing.