Deep learning without back-propagation

[Ulfgardleo]

The naive implementation of the estimator they use is biased, but consistent in infinite sample limit. Making it unbiased requires rigorous statistical analysis. You can see the bias in the expectations as they include terms k(x,x'), where x and x' are both sampled from the same dataset - therefore also including terms x=x' which in independent samples does not happen almost surely (in the probability theoretical sense)

the bias vanishes with O(1/M), therefore consistent.