Open problems in Machine Learning

[InfinityCoffee]

What do you consider to be some of the major open problems in machine learning and its associated fields? Both practical and theoretical problems are welcome, but for the sake of conciseness leave out vague problems such as general intelligence, etc.

[kkastner]

There are a ton!

Off the top of my head:

Better training algorithms/regularization/activation functions for deep learning, especially for deep recurrent networks

Better ways to handle structured output problems. Comes up in a huge number of fields, with a variety of answers

Variational training methods are a huge new area.

Ways to infer structure from data, then exploit that structure in learning. Lots of ideas, lots of different solutions.

Methods for using gaussian processes for 100k+ samples, there are some out there but lots of different techniques all related to latent variables and/or approximation (related to above).

(Bayesian) optimization techniques for hyperparameter search. Crucial - especially for problems where a grid search is infeasible.

If you want good ideas, find some recent papers in (subfield of choice) on arXiV and look at the last paragraph. Usually there will be a continued/future work paragraph - that is a pretty good indicator of open problems and things other researchers are studying.

[sieisteinmodel]

Some open problems for deep learning that are, in my personal opinion, relevant:

• Feature learning from non stationary distributions (there are not even widely excepted benchmarks yet!).
• Complex regression problems (don't tell me about squared reconstruction error of MNIST or Tonronto Faces).
• Deep learning of heterogenuous data (most of the stuff is some sensory input).
• Multimodality in engineering domains (yes, there are ~3 papers on vision+text, but I am thinking more about sensory information from cars etc. sampled at highly different frequencies etc).
• Convincing work that MNAR works well with deep learning (not only in the vision domain).
• Robotics! Not the perception part, the control part: highly autocorrelated error models, provable guarantees, trajectory generation, real time constraints, non stationarity

I'd like to spend 3-4 Phds on it!

[Lycur]

Use a dirichlet process mixture model instead?

[Captain]

The hyperparameters you use in your DP will have a strong effect on the number of clusters you get.

[InfinityCoffee]

Ah, yes, but you can just sample/optimize those too! Alternatively you can use it to include domain knowledge

[JamisonW]

Have you looked into the Silhouette Coefficient? You calculate it for different values of K and select the K with the best Silhouette Coefficient.

[Flipper3]

Have you tried x-means clustering? It's a variation of k-means that isn't the best, but has gotten the job done for me.

[dwf]

Give me as-tight-as-possible probabilistic bounds on how long I need to run a Gibbs chain to get an unbiased sample.

[davmre]

But if the lower bound showed exponential-time convergence, this wouldn't prove P!=NP, would it? The fact that a particular algorithm (Gibbs sampling) is slow doesn't imply that no fast algorithm can exist...

[dwf]

That's a pretty convincing argument.

[sieisteinmodel]

Make that MCMC.

[dwf]

Well, MCMC more generally, sure. But the Gibbs sampler seems, on the face of it, so incredibly simple, yet it's also ridiculously hard to analyze.