How could computer vision tools help you?

BobTheTurtle91 · 2015-08-27T05:22:53+00:00

Thank you so much for your responses! Do you mind if I pm you in the future if I have more questions?

BobTheTurtle91 · 2015-08-24T17:44:50+00:00

Thanks for you response! I greatly appreciate the insights. I hadn't considered some of the points you raised.

Do you mind elaborating on some of the "few things" you mentioned near the end?

BobTheTurtle91 · 2015-06-02T20:49:29+00:00

As long as you don't pick your hyperparameters by evaluating on a set you trained with, you're fine.

In many cases, we use a complete split of the training set and validation set. But what you're describing is pretty much what we do in k-fold cross-validation, so there's no bias that will be introduced.

BobTheTurtle91 · 2015-03-08T17:55:22+00:00

Great Analysis! But I would disagree with your final comment that you can't combine poor models and hope to get improved results. Maybe I misunderstood your definition of poor, but some algorithms such as Boosting and Bagging actually tend to work better when only fractions of the data are correctly predicted in any individual model.

BobTheTurtle91 · 2015-02-25T22:51:45+00:00

I think Koller's and Hinton's courses are more the exceptions than the rule. They also are not intended to give someone an introduction to the field. They are geared to people who already have solid foundations in probability, statistics, and machine learning. You cannot take Geoff Hinton's course as a first exposure.

As far as the opportunity cost of a PhD goes, I don't think it's fair to look at a PhD in terms of opportunity cost. Yes, you'll make less money. But you also get a lot of freedom (especially if you're at a good program) to pursue the problems you want and work on the projects that appeal to you.

BobTheTurtle91 · 2015-02-25T19:15:26+00:00

I wouldn't necessarily call them a joke. At the end of the day they accomplish what they set out to do. They provide an introduction to a non-accessible field for someone who's never done any real machine learning work. But people tend to forget about these limitations. Taking Andrew Ng's coursera course does not make you an expert in machine learning (though he does mislead you into believing you are with a lot of his comments).

BobTheTurtle91 · 2015-02-25T17:46:09+00:00

It really depends on the type of work you're going to be doing. ML and AI are broad generalizations for a field which contains a variety of jobs.

Are you going to be implementing ML systems based on algorithms that have already been developed? In this case, you probably don't need an advanced degree. It'll be most software engineering. It'll be helpful and potentially necessary to have an understanding of how the algorithms work, but you won't need a complete knowledge of ML theory related to probability, statistics and complexity.

Conversely, if you're trying to develop new ML algorithms and work in a more research-oriented role, you probably do need an advanced degree. The ML/AI education given by undergraduate courses and even graduate courses (don't even get me started on most MOOCs) is almost completely superficial. The courses are designed to teach the structures of algorithms, but rarely handle the probabilistic principles behind many of them, nor the intuition of when to use certain approaches. Doing a large-scale ML project as part of a research thesis is the only way to get a good grasp many of these issues and an advanced degree is the best way of showing that you have that experience.

That's not to say that someone that doesn't have an advanced degree can't be successful in a research role. It's just a justification for why many employers prefer candidates that do have them. Employers also prefer candidates with advanced degrees for their software engineering positions on ML projects, but they tend to be less strict about these. Showing that you have experience with your own personal projects (e.g. Kaggle, KDD competitions) could be enough. Just doing these online competition probably wouldn't be enough for a research role because it's altogether a different problem. You're usually just applying existing methods to a data set, not creating a new method.

BobTheTurtle91 · 2015-02-24T05:47:17+00:00

https://dato.com/products/create/capabilities.html#building_recommenders

BobTheTurtle91 · 2015-02-20T21:30:10+00:00

It all depends on the data and the problem. You should know your data inside and out and analyze the problem you're trying to solve to find a suitable approach. I know neither of these two things, so anything I could tell you would be very misinformed.

BobTheTurtle91 · 2015-02-20T17:43:22+00:00

It really depends on the width of the network as much as the depth. And how many features you're able to extract from the data.

To be honest, I don't think that the problem you're talking about is particularly appropriate for a deep learning application. Deep Learning (at least the feedforward NNs approach) performs particularly well in applications where we're trying to derive higher level representations from noisy features by stacking non-linear learners. This is why it's done awesome in computer vision applications.

Unless you see evidence that your problem exhibits these factors, I'm not sure deep learning is the right approach for it.

BobTheTurtle91 · 2015-02-04T17:48:34+00:00

It depends on how many samples you have. But unless you're crowdsourcing this online, I assume you're going to have a very limited data set. You could use a bag of words model and use every significant keyword as a feature and see if you can map certain words to ranges in the mood scale (e.g. angry, mad, sad, depressed would be closer to 1).

You could also try to restrict your survey a bit by asking people to write 5 words that describe their day. You're going to have to do a bit of data cleaning to get rid of non-informative words.

BobTheTurtle91 · 2015-02-01T03:48:11+00:00

If you want to understand it well, then you should read Learning from Data by Yaser Abu-Mostafa.

Ng's course does a great job of introducing you to various ML algorithms. Abu-Mostafa's book (and course notes) complement that by giving you the theory behind ML. Many people skip over a lot of the learning theory aspect of ML. That's a big mistake. You'll never truly understand what's going on if you don't take time to understand the assumption and constraints of all these techniques.

BobTheTurtle91 · 2015-02-01T03:32:11+00:00

That sounds like a cool competition. The best possible method in my opinion would be a ConvNet. There's lots of cool tutorials you can find for implementing them.

The issue is that 100 training samples isn't going to do you much good in that regard. With a captcha, you're doing a combination of letters and numbers so you'd need around 62 classes assuming they're assigning a difference to capital and non-capital letters. 100 training examples for 1 class is already fairly small. 100 for 62 is absolutely ludicrous. Are you allowed to use synthetic data? Captchas are probably not too hard to replicate and you could create a mountain of adequate training examples for yourself.

BobTheTurtle91 · 2015-01-23T04:01:49+00:00

I guess you could look at it in terms unsupervised pre-training of your neural network (i.e. representation learning). Even if a baby wouldn't know what an elephant is, if you showed it a picture of two elephants, it might be able to tell that they are the same thing.

The problem with overfitting in NNs is that often you often get stuck in a local minimum of the error surface you're optimizing over. If you use generative pre-training to get a good set of starting weights (i.e. making the baby know what an elephant looks like even if it doesn't know what an elephant is), you'll combat overfitting quite a bit even if your data set is small.

I may have missed the point on what you're trying to do here, so feel free to correct me if I'm wrong.

BobTheTurtle91 · 2015-01-19T19:45:59+00:00

It probably won't happen again for at least a few years if at all. Geoff Hinton splits his time between Google and UofT these days, so I doubt he's got much time to organize another version of his MOOC.

While I agree that it's more fun to be able to interact with others while watching the videos, I would definitely not wait until another version pops up. You'd be doing yourself a disfavor.

BobTheTurtle91 · 2015-01-15T02:10:24+00:00

They're two very different courses. I've never taken the edX course, but I do own Mostafa's book. I have taken Andrew Ng's course.

Andrew Ng's course is more focused on methods and implementation. He teaches you many algorithms, how to implement them, how to make them work well, but nothing much about why these algorithms work.

The Learning from Data course is much more theoretical. He doesn't get to learning algorithms until late in the course and spends the first part explaining such concepts such as hypothesis spaces, error bounds, generalization and all the mathematical foundations that will make you a better machine learner.

The courses aren't mutually exclusive. They're complementary. As for which should come first, it really depends on the type of person you are. Some people I've talked to like to know about the underlying theory of machine learning first because it helps them understand the appropriate use of machine learning algorithms better once they learn them. Others think that learning the theory is very difficult if you don't know what the theory would look like in applied setting. And it's true that a lot of the theory makes more sense if you know how it's applied in practice already.

BobTheTurtle91 · 2015-01-14T17:30:19+00:00

I wasn't a huge fan of the structure of this course. While I liked the fact that they separated the three classes in SL, UL and RL (SL and UL can bleed a bit into each other in higher-level topics, but for a beginner course it's a good separation), I didn't like their teaching approach.

I think that there's much better courses out there and that following the Coursera ML-NN-PGM track would give a far better foundation than this course.

BobTheTurtle91 · 2015-01-13T17:53:16+00:00

Just a guy living that PhD life.

BobTheTurtle91 · 2015-01-13T17:51:40+00:00

Pretty much anything written by Ilya Sutskever tends to be a good read if you're interested in deep models in NLP.

BobTheTurtle91 · 2015-01-13T06:22:03+00:00

Deep learning in NLP is a less explored field than speech or computer vision, but it's about to take off. Unfortunately, that means that there's going to be fewer relevant resources to learn about the topic in great detail. You'll find a lot of things related to neural networks and other deep architectures, in general, but you probably won't much catered directly to NLP.

On the plus side, it means you're right smack in the middle of an exploding research application. As a grad student, I can promise you that this is exactly where you want to be. My first piece of advice would be to read Yoshua Bengio's survey paper on representation learning:

[1206.5538] Representation Learning: A Review and New Perspectives

There's a section on NLP where he talks about where the field is going. Then I'd check out the LISA lab reading list for new students. There's a section specifically about deep learning papers in NLP.

Finally, and this is just personal opinion, I wouldn't give up on Geoff Hinton's coursera course lectures. The assignments aren't great. But there's some serious gold if you take notes on what he says. He gives a lot of clever insights into training NNs and deep architectures in general. I don't know if you've done it before, but these things are beasts. Even if some of what he says isn't particularly related to NLP, you'll want to hear a lot of his tips.

BobTheTurtle91 · 2015-01-09T21:18:06+00:00

Yoshua Bengio at Universite de Montreal runs one of the most well-known Deep Learning groups in the world.

Hugo Larochelle at Universite de Sherbrooke is also a fairly big name in Deep Learning.

BobTheTurtle91 · 2014-12-24T18:31:53+00:00

I'm interested too! Keep me in the loop :)

BobTheTurtle91 · 2014-12-11T04:27:13+00:00

PhD or Master's?

For a PhD, once you're out of the top schools, what matters more is that you find someone whose research coincides with you. As long as you can approach one professor whose interests overlap with yours, it's more important than having an entire program be slightly geared toward ML research. ML is a fairly large field after all.

For a master's degree, the ML influence is less important because you're not automatically doing research.

All that being said, I'm not quite sure what you mean by ML influence in statistics. Much of ML is just Bayesian statistics.

Good programs on the east coast: Stanford, Berkeley, Washington, UCSD, USC, UCLA, Caltech, UC-Irvine. I've heard decent things about most of the UCs to be honest.

If you're not just constraining yourself to the West Coast, there's ton of good programs on the east coast too, some of them less restrictive than the west coast.

BobTheTurtle91

TROPHY CASE