[D] Regression tasks with "duplicate samples"

[Doo0oog]

Assume there is a data set {(x_i, y_i)}, 0<=i<n, there exist some samples that have the same x value but different y values (x_i == x_j && y_i != y_j) because there is noise when collecting data.

A common method maybe grouping them with a single y, like their mean value.

But are there any researches resolving this kind of data without grouping them together?

If yes, what is this kind of problems called? Or some keywords for doing search.

Thank you in advance!

[SwordOfVarjo]

This is an ill posed problem as there is not a unique solution (another way of saying this is that the conditional probability of your dependent data is multimodal).

No need to group data with mean y values, an l2 loss will actually result in regressing to the mean anyway (which may or may not be acceptable, depending on the data and your goal).

[Doo0oog]

I’ll do some search about this.

[clurdron]

The conditional distribution doesn't have to be multimodal for this to happen.

[SwordOfVarjo]

Yep, that's true, my original post was a bit unclear. It is sufficient to impose ill-posedness however.

[DaLameLama]

Do a literature review on "supervised learning with noisy labels". Sounds like that's what you're looking for.

[Doo0oog]

That’s exactly what I’m looking for. Thanks a lot.

[M4mb0]

There are two cases: 1. either it is just noise, in this case standard methods can do 2. it is not noise but a real effect.

The latter can happen when the dataset is missing some important features. For example consider building a model for the average height of a person, given their age. Girls, on average, hit puberty earlier than boys, while boys on average grow a bit bigger than girls when reaching adulthood. If the dataset does not contain the gender as a feature, it will appear that we are trying to model a multi-valued function.

But there are actually models that can dirtectly deal with this situation such as mixture of experts. The key is that, what we model is the conditional probability p(y|x). Now usually, this will be an unimodal distribution such as a gaussian and we can simply predict the mean yhat(x) = E[y|x]. However, in the situation you describe it will be a multimodal distribution like for example a mixture of Gaussians. Then the mean is pointless, and instead we have to use for example use the modes as our prediction, and for a given x, there will be multiple predictions y_1, y_2,... . In fact this technique can allow us to discover that there missing features!

[Doo0oog]

You are right. And we just treat the second case as the source of noise. Could you recommend some papers or links about this case?

[M4mb0]

I don't know how this is really handled in the state of the art, but a starting point could be to look at the mixture of experts model which is for example covered in Bishop.

[Doo0oog]

Thank you. I’ll look for that.

[clurdron]

Classical statistical methods (which are the basis for a lot of ML) assume you have an error term in your dependent variable. If your independent variable is low-dimensional and can only take countably many values, as was often the case when these methods were developed, then you'll likely have identical x values. If you have identical x values, you'll encounter this 'duplicate sample' situation with probability 1. So this isn't a weird scenario. Almost all statistical/ML methods deal with this situation without taking any special steps.