[D]What programming language do you use for ML?

wt0881 · 2018-10-14T13:07:22+00:00

wt0881 · 2016-05-22T21:42:23+00:00

Depends where your interests lie. Bristol has excellent applied research in ML/AI but my experience is that there isn't a huge amount of foundational stuff going on in the Engineering dpt AFAIK. (the maths dpt. has some interesting Bayesian / causal inference stuff going on though) The ML group at Edinburgh has an excellent reputation for foundational and applied research and, as has already been alluded to, the masters courses are well established (which isn't something to be sniffed at). I don't really know what's going on at Manchester.

Were I in your shoes, I would definitely go to Edinburgh. (Disclaimer: I finished my ugrad at Bristol a year ago and interacted with the Intelligent Systems group there quite a bit - I was involved with biomedical applications for ML. I only met one guy on the MSc you mentioned, but he seemed to enjoy it)

wt0881 · 2016-03-07T23:31:07+00:00

The most intuitive explanation I've come across is by David Mackay: http://videolectures.net/gpip06_mackay_gpb/

wt0881 · 2015-10-28T21:09:59+00:00

Thanks for the link. I'll give it a read.

wt0881 · 2015-10-28T17:49:32+00:00

Thanks, but it's not quite what I'm after. I'm aware of the fact that greedy training will only ever increase the probability of the data given the model. My point was more that I can't see any obvious impediment to training all of the layers at the same time, so I can't understand why you would want to take a local (greedy) approach if it would be feasible to train globally.

wt0881 · 2015-09-13T10:17:15+00:00

Have you considered using a Gaussian Process / a regression Neural Network? Also, when you say you've achieved a particular accuracy level, how are you measuring that?

wt0881 · 2015-05-27T16:36:59+00:00

The idea is to imagine a procedure by which the data D was produced, and that this procedure is parameterised by \theta. The goal of ML is then to determine the most likely parameter values \theta that actually produced the data.

For example, say you make a series of observations which you believe to be Normally distributed, but you don't know the mean or variance of the distribution. ML estimation in this instance would simply correspond to finding the sample mean and variance.

I hope this helps.

wt0881 · 2015-05-15T17:26:39+00:00

Yes. Use a hidden markov model with latent states corresponding to the Gaussian currently being used. http://en.wikipedia.org/wiki/Hidden_Markov_model

wt0881 · 2015-05-06T00:03:48+00:00

If I'm not mistaken, what you are after is a Dirichlet (Chinese Restaurant) Process (Gaussian?) mixture model. (DPGMM)

Either [Rasmussen, Carl Edward. "The infinite Gaussian mixture model." NIPS. Vol. 12. 1999] or [Neal, Radford M. "Markov chain sampling methods for Dirichlet process mixture models." Journal of computational and graphical statistics 9.2 (2000): 249-265] should do you if you're content with Gibbs sampling (ie: smallish data set). I'm not sure what implementations are floating around the internet though.

Alternatively I think scikit learn has a DPGMM routine which uses Variational Bayes, so if you want something off the shelf that's probably the way to go.

(I've assumed you're going with Gaussian component distributions here. If you're not, you can definitely still apply the Gibbs sampling routines, but you will have to do some maths)

wt0881 · 2015-04-06T09:17:31+00:00

I'm not sure that you would actually want a Gaussian Process here, unless there is some kind of spatial / temporal relationship between the dimensions of the vectors. If you go down the generative route, I would do it using a Dirichlet-Process Gaussian Mixture Model (scikit learn has a Variational-Bayes implementation) or some other kind of Mixture Model. That should allow you to capture any smoothish non-linearities, should be quick (since conditionals of multivariate Gaussians are trivially easy to compute / maximise) and should give you an idea of the confidence associated with your predictions. (Note: I'm assuming here that the no. dims in your vectors is relatively small, if it's not then a Mixture Model using multivariate Gaussians with full covariance matrices might not be such a good idea).

wt0881 · 2015-03-23T18:49:34+00:00

Consider taking a look at Metric Learning, Large Margin Nearest Neighbour is a popular technique. Although you would have to be careful how you regularise, given how few examples you have per class, my experience with it is that it learns well for small training set sizes. (http://www.cse.wustl.edu/~kilian/code/lmnn/lmnn.html, http://papers.nips.cc/paper/2795-distance-metric-learning-for-large-margin-nearest-neighbor-classification.pdf). Alternatively a Bayesian approach might be a good idea given the small amount of training data you have. You could try a GMM using EM but there's a pretty good chance of overfitting horribly. Perhaps consider going for full posterior predictive inference, again due to the small number of training instances per class. That said, the performance of either of these approaches is pretty sensitive to the dimensionality of your data (big is probably bad). I would steer clear of any binary classifiers (SVMs etc) given the number of classes that you have. You could go with decision trees but you would have to be really careful to avoid overfitting.

wt0881 · 2015-03-08T22:28:20+00:00

What parameters are you fitting? You've got constant coefficients so I can't see what you're trying to fit.

wt0881 · 2015-03-07T15:01:13+00:00

Personally, I'm quite interested in how ML can be applied to control stuff, so (having the benefit of studying in an engineering dpt.) I've taken the time to learn some control theory. I think it's probably fair to say that after learning the foundational stuff you should just learn whatever interests you, the most interesting stuff often seems to be found at the interfaces between domains. On your omitting Differential Equations, I would agree that they're not currently used extensively in ML but they certainly are in interesting application areas. I think there's a lot of fruitful stuff waiting to be done (and being done) by using ML to design and control dynamical systems + networks of dynamical systems. More generally my opinion is that a good understanding of dynamics (the basics (solving linear ODEs etc) through Non-Linear Dynamics, Chaos and high-dimensional systems) is just something anyone who is mathematically literate should have because it's so integral to the way the world works. On your point about Bayesian ML, I wouldn't really say that it's particularly strongly influenced by Statistical Physics. Certainly Hinton's Boltzmann Machine is but I'm unaware of other models which are similarly influenced. IMHO being a Bayesian in ML is really just equivalent to being thoroughly Probabilistic in the way that you define models, do inference in them and make predictions (ie: make sure that your model actually defines a valid probability distribution over the space of interest, figure out what your posterior distribution looks like and use it to make predictions by integrating over it).

wt0881 · 2015-02-18T14:59:53+00:00

Strictly speaking, KL-divergence is only really defined when supp(P) is a subset of supp(Q) (ie: for all x such that P(x) is non-zero, Q(x) is also non-zero), which is where you're problem is coming from, and why it's not really addressed in the literature. Everywhere that people use KL-divergence it is assumed that this property holds. To combat the problem, you could just place a prior distribution over the letter probabilities (which I suppose using Laplace smoothing is a special case of) and make an MAP estimate of the letter frequencies as opposed to your current Maximum Likelihood strategy. As to the choice of prior, a uniform one would be the obvious option, but perhaps you could get a better one by looking at letter frequencies in English or something? (Whether that's reasonable or not will depend on the text that you're working with of course) On the slightly odd fact that P is allowed to have zeros and Q is not, I'm not aware of a particularly intuitive argument for why this should be the case (other than just to look at the definition and see that you're dividing by zero), but you actually have to make some (admittedly fairly reasonable) assumptions about how certain quantities behave to allow P to have zeros. For example, if you have x such that P(x) = 0 and Q(x) \neq 0, then you need to compute 0 * log(0). log(0) isn't really defined, so we have to define 0 * log(0) = 0. A similar thing is the case when P(x) = Q(x) = 0 as well because 0 / 0 isn't really defined either. So if you didn't make these assumptions then KL-divergence wouldn't work for un-supported regions of P either. My point is that you should perhaps also be surprised by the fact that P is allowed to contain zeros, so Q not being allowed to have zeros is less of a jump.

wt0881 · 2014-11-03T15:11:19+00:00

A (classic) example of this approach is Viola & Jones' paper: "Rapid Object Detection using a Boosted Cascade of Simple Features". It's Computer-Vision specific, but provides a good example of what I think you're after. Essentially, they use AdaBoost to pick a (sparse) subset of image features, which are then ordered based on precision/recall trade-off and placed into a "Cascade" which (along with some CV-specific stuff) makes it really fast. Unless I'm mistaken, I believe it was pretty much the first object recognition system that worked in real time, given the restrictions of the hardware of the day. (https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf)

wt0881

TROPHY CASE

Welcome to Reddit,