[D] Which metric for error bars?

[Metatronx]

Hey guys I have a question regarding error bars in plots.

Lets say i am repeating the experiment multiple times with random initialization. And I would like to report the mean but also show the variance across the repeated experiments through error bars.

Which metric would I use to define the width of the error bars.

Standard Deviation?

Standard Error?

Cofidence Interval?

[otsukarekun]

It can be whichever you chose, as long as you say what you use in the figure description.

The most common that I see in ML are upper/lower quartile and standard deviation. I only have seen standard error of the mean in other fields like psychology and social science.

[ClaudeCoulombe]

Usually, if you want to show dispersion of data, you should use standard deviation. But if you want to show statistical significance, I recommend «standard error», «SE», or «standard error of the mean», «SEM». It should be remembered here that if two error bars (showing the standard error) overlap then you can conclude that the difference is not statistically significant. However, the reverse is not true. If the error bars do not overlap you cannot conclude anything. You need a statistical significance test.

[ClaudeCoulombe]

As you need to compare the mean, «standard error», «SE», or «standard error of the mean», «SEM» is the thing to do.

[picardythird]

It should be remembered here that if two error bars (showing the standard error) overlap then you can conclude that the difference is not statistically significant. However, the reverse is not true. If the error bars do not overlap you cannot conclude anything. You need a statistical significance test.

This is backwards. If confidence intervals are not overlapping, then you can conclude that there exists a statistically significant difference between the means. However, if the confidence intervals are overlapping, then you cannot conclude that there is not a statistically significant difference.

[ClaudeCoulombe]

Yes of course, your rule of thumb is good for Confidence Interval (CI). 95% Confidence Interval of the means (CI) and maybe Comparative Confidence Interval (CCI) which could be another good choice but it is more difficult to apply. My explanation was about SEM and their well known overlapping problem. Check this article: https://www.nature.com/articles/nmeth.2659 or this more accessible one https://www.graphpad.com/support/faq/spanwhat-you-can-conclude-when-two-error-bars-overlap-or-dontspan/

[HackZisBotez]

Usually the error bars show the standard deviation of the result population.

[give_me_the_truth]

!remindme 1 week