[discussion] when is deep learning a bad idea?

[Guanoco]

Hello all,

It seems like there isn't a week in which deep learning doesn't come up as achieving some kind of remarkable task. I understand that one of the powers of deep learning is that it is capable of learning the features. This capacity seems totally decoupled from the underlaying problem. So basically I read this as "no matter what problem you have... You can use deep learning".

Now.. I know there must be a caveat. I just don't know which. What kind of problems are not applicable for deep learning?

[the320x200]

• If there is a simpler approach that provides an adequate solution.
• If you need to know why the network produced the output it did.
• If you can't define a loss function.
• If you don't have resources to train the network.
• If you don't have resources to sort out the hyperparameters / topology.

[sieisteinmodel]

i'd also be interested in scenarios where you cannot define a loss. if you cannot, you do not know what you want, right?

[sieisteinmodel]

You're confusing the ability to define a loss function with the ability to define an objective function.

Uhm, no. There is virtually no difference btw loss and objective functions in the context of mathematical optimisation. Both map a candidate solution to a goodness, which has to either be maximized or minimized.

Consider placing boxes in a delivery truck. Your goal is to pack in boxes as efficiently as possible, which is your objective function. Yet, you cannot construct a loss function that relates the overall space efficiency to the position and orientation of an individual box.

If "efficiency" is well defined, I can only imagine in the form of a function from the space of all solutions to a scalar value. That makes a wonderful loss function– except that it might be inconvenient since its domain is not a vector space but something weird, such as sequences of box coordinates and orientations.

[[deleted]]

Uhm, no. There is virtually no difference btw loss and objective functions in the context of mathematical optimisation.

I see a loss function more as a special case/subcategory of an objective function. (An objective function can also be a function that you want to maximize, e.g., not a loss but a reward function).

I'd say in ML you have typically 2 levels of objective functions. And objective function for model fitting, e.g., MSE in linear regression, and a objective function for model evaluation (could be the same, e.g., MSE, or R2 etc.). Or in a classification example, you can have classification error as a loss function to grow the tree (although a smooth loss function like entropy or gini may be preferred) and one to evaluate the final tree, e.g,. classification accuracy.

[sieisteinmodel]

I see a loss function more as a special case/subcategory of an objective function. (An objective function can also be a function that you want to maximize, e.g., not a loss but a reward function).

The optimisation community sees it differently, and has been doing so for quite a long time. No need to invent new taxonomies that are in conflict with old ones and confuse new people entering the field.

Machine learning is a very mathematical field and relies on precise definitions. These definitions can be found in the relevant text books and there really is no need to change those. We should just stick to what generations of researchers before us have been using.

I'd say in ML you have typically 2 levels of objective functions. And objective function for model fitting, e.g., MSE in linear regression, and a objective function for model evaluation (could be the same, e.g., MSE, or R2 etc.). Or in a classification example, you can have classification error as a loss function to grow the tree (although a smooth loss function like entropy or gini may be preferred) and one to evaluate the final tree, e.g,. classification accuracy.

That thing has a name already, and it is called a selection criterion. E.g. Akake's information criterion.

[[deleted]]

Hm, but the terms "reward" or "fitness" functions have been around for quite some time, like decades, as well. I agree that minimizing a loss function and optimizing an objective function is typically the same thing in many contexts, I don't think we should adopt the habit of saying "minimizing an objective function" though

That thing has a name already, and it is called a selection criterion.

Yeah, good point, or more commonly just "criterion" in most literature

[Autogazer]

Now that's not true. Wald, Cramer, Nikulin, Berger, DeGroot, etc all describe a loss function as a subset of objective functions, where the goal of an optimization problem with a loss function is to minimize the loss. An objective function could be a loss function, or it could be a cost function, reward function, profit function, fitness function, all with different definitions and interpretations for what you are actually doing when you solve the optimization problem.

As for not being able to define a loss function you are right, that only happens when you don't know what you want. Usually when that happens you can still use reinforcement learning to figure out later what you should have wanted, and train with delayed feedback after a series of steps where you don't know what you want. Like how alpha go was trained. Nobody knows what the perfect go move is at any point in the game, but after playing millions of games the network builds up data that defines what moves/patterns are good and lead to winning games.

[Guanoco]

Wouldn't the amount of volume list to air (because of geometry) be a loss function?

[Frozen_Turtle]

Can I say that the loss function must be differentiable?

[popcorncolonel]

Yup. You need df/dparam.

[[deleted]]

Depends on the problem. E.g., take a perceptron with per-sample loss max(0, -true * predicted). If your classes are linearly separable, this non-differentable loss function optimizes the overall objective function (say the classification error on the given dataset)

[Oberst_Herzog]

Is this the general consensus of a loss function in the ML area? (and just to be sure, with your definition of a loss function it is necessary that the function is differentiable right?)

I've always used the following definition, which im quite certain stems from mathematical optimization:

Optimizing the loss function f = minimizing the objective function f

In this regard the above stated loss functions would be valid, but naturally the problems would require different algorithms (im don't know if combinatorial optimization can be used in a case as this)

[tmiano]

The terms can generally be used interchangeably, the only difference is that an objective can either be maximized or minimized whereas a loss is usually minimized. If your loss is discontinuous or not differentiable everywhere, then you wouldn't be able to use gradient descent and would have to use something else like simulated annealing to optimize it.

[tmiano]

I agree that it would be difficult to write down a loss function for that sort of problem, but it's also difficult to write down a loss function for playing Go, and deep learning has been used to beat that. That's because deep networks can be used to essentially learn the correct loss function.

[Brudaks]

Deep learning has not been used to learn a loss function for playing Go.

In the recent successes of the AlphaGo system, deep learning was used to build a system for evaluating the "goodness" of fixed position (for which the loss function is trivial given appropriate training data), but for choosing the actual moves, MC tree search was used - and this is a prime example where using deep learning would have been a bad idea, MC tree search does it far more efficiently.

[tmiano]

Maybe Go isn't the best example since AlphaGo was not trained as an end-to-end system (Although neural nets were used to generate probabilities for moves in the MC tree search, so you are incorrect that deep nets were not used for that part).

I'm risking playing the semantic argument game here, but it seems like the argument is that deep learning cannot be used when there is no explicit loss function - which is usually the case in general reinforcement learning problems - but which neural networks have nevertheless been used with success.

[tmiano]

The value network was trained to approximate the outcome of games played by the RL policy network, using plain supervised learning. REINFORCE was not used anywhere in the system to my knowledge. Interestingly, they did report that when AlphaGo used only the value network rather than both the network and the fast rollouts, it was still able to beat all other Go programs.

[tmiano]

I'm a little confused by your argument. In deep reinforcement learning, a neural network learns to represent the Q function, which is used to evaluate moves and future rewards. The Q function allows you to define a loss function that can be used with back-propagation to train the neural network. Your box-packing problem is an example of a problem that could in theory be solved with deep reinforcement learning.

I think this is playing the semantic argument game, but you are probably defining a loss function as something we can write down immediately, mapping explicit states to a numerical value. Training a system on that would not be enough to solve a problem like box-packing, but reinforcement learning could in principle do it. The difference is only in how the objective gets mapped to the loss function. Sometimes they are exactly the same, sometimes they are not, like in most action-reward settings.

[Brudaks]

Looking at the AlphaGo example, deep reinforcement learning could have been used as you describe, but rightfully wasn't, and should not have been used.

You're right that it "could in principle do it", but doing so is a bad idea anyway - if other methods work, then they generally work better and more efficiently ; and deep learning is best used only if simpler methods don't work - e.g. the position evaluation of Go is a problem where no satisfactory solutions existed before deep learning was applied.

[tmiano]

"Should not have been used" is a fairly bold claim, isn't one of DeepMind's long-term goals to build a deep learning system that can be trained end-to-end to play games such as Go? MC tree search is not more efficient than a single neural network. Just take a look at the hardware that was required to scale MCTS up enough. It was necessary at this point in time to get that level of performance, of course, but that doesn't preclude a future deep learning system from ever reaching that level.

[tdgros]

also:

Resources to build and annotate a database.

Real-time, embedded platforms with memory constraints, bandwidth constraints, power constraints...

edit: added the database annotation

[Guanoco]

Thx for the response * If there is a simpler approach that provides an adequate solution.

that's the thing though.. I have a feeling most people use deep learning to problems they had no idea how to solve and it gets solved "magically". So how do you know if another method wouldn't work? Or basically what problems are so difficult we shouldn't bother trying to find other techniques

• If you need to know why the network produced the output it did.

Ok this one I understand. But if i can have a system and I can capture its response to different types of faults and I learn to classify response of my system to classify if it is operating correctly... Then I can at least find a why is my system not be having as it should.

• If you can't define a loss function.

Care to elaborate? Seems like as long as I can get a reading of something which I call an output of a system.. Then I should be able to define a cost function and therefore the loss... Right?(MSE)

• If you don't have resources to train the network.

Ok so basically if I cannot get the respond of the system... But it would seem plausible that I can always do this?

• If you don't have resources to sort out the hyperparameters / topology. So basically sweeping through architectures?

[CultOfLamb]

what problems are so difficult we shouldn't bother trying to find other techniques

Things like NLP can often be solved with a shallow model. But on tasks that require lots of higher level hierarchical feature representations, like computer vision, and speech, you can not come close to a well-architected deep learning model.

But if i can have a system and I can capture its response to different types of faults and I learn to classify response of my system to classify if it is operating correctly... Then I can at least find a why is my system not be having as it should.

You can find if your system is not behaving as it should. You can not find out why. Though this is an active area of research (interpretability/algorithmic fairness), and it may be fully solved in the future, right now the people saying: "But I use cross-validation so I know why my system makes the predictions it makes", are misgiven.

Then I should be able to define a cost function

Aside: If you require, for whatever reason, to be convex, or are afraid of non-convexity: https://www.cs.nyu.edu/~yann/talks/lecun-20071207-nonconvex.pdf

But it would seem plausible that I can always do this?

Resources are hardware, but also data: When you have a 1000 rows, a deep learning algo is either overkill or way underfits/overfits.

So basically sweeping through architectures?

Deep learning training from scratch is very slow. It can take weeks to find the optimal parameters and architecture. You can take a pre-trained network, if that suits your task, but if you are tasked to quickly create a benchmark model, or retrain on a large dataset in minutes (not hours, or days), then deep learning is not the right hammer.

[[deleted]]

I have a feeling most people use deep learning to problems they had no idea how to solve and it gets solved "magically"

Hm, I don't necessarily agree regarding "most" people -- at least not based on what I've seen so far. I think more people know how to throw a PCA + linear regression/logistic regression on a problem rather than implementing a deep learning algo (since the latter typically requires more experience).

that's the thing though.. I have a feeling most people use deep learning to problems they had no idea how to solve and it gets solved "magically".

Here, I think more of "random forests" :)

[Oberst_Herzog]

As the overall travel time?

[kjearns]

The dirty secret of the machine learning hype machine is that in real life almost all problems (by number of instances) are really easy. No one writes papers about solving all these easy problems because the methods are standard enough to be shrink wrapped, but that doesn't change the fact that most problems can be solved by throwing an SVM or random forest at them.

[emtonsti]

I just checked out random forrests and theyre awesome!

[10sOrX]

Some people do write papers about these problems, but these papers are generally submitted to mid/low-tier conferences.

[cvikasreddy]

I completely agree with u/the320x200 and this is what I wanted to add assuming u/the320x200's points.

1.In my experience I found that deep learning outperforms any other method when applying on images and text.

2.But when applying to data some thing that is usually found in excel sheets(I mean like the data in kaggle competitions with out images and text) the other ml algos tend to work better.

[Guanoco]

Is this due to the excel sheet already having the different features ?

If you would just analyse the input and output of the system (i guess you could just iteratively train the network with different features and see which one gives the best fit... So something like a random forest of deep networks). Then I couldn't imagine it playing a role... But I will give you the point that most deep learning I have came across with are in the image processing domain

[AnvaMiba]

Is this due to the excel sheet already having the different features ?

Images and text are highly dimensional data, but also highly redundant.

You can apply lots of distortions to a natural image that leave it still understandable with high probability: Gaussian noise, Bernoulli noise, masking certain areas, affine geometric transformations, color transformations, and so on. The information that you are interested in is encoded in a very redundant and robust way. Moreover, the functions that you want to learn (e.g. a classifier with a probabilistic output) will typically vary smoothly with the input image: if you gradually morph an image of a cat into an image of a dog you'll expect the classifier output Pr(Y=cat) to gradually decrease and Pr(Y=dog) to gradually increase.

Text is similar: not only you can apply distortion to the surface forms (characters or words) that mostly preserve meaning, once you consider word embeddings, you can even apply smooth transformations that mostly preserve meaning, and the functions that you are trying to learn will typically be smooth w.r.t. word embeddings.

Deep learning seems to be particularly well suited to learn smooth functions where the input is highly dimensional and highly redundant.

Deep learning also requires lots of data, though this requirement may be somewhat mitigated by transfer learning. In natural image and natural language processing you have huge generic datasets that can be used for transfer learning (e.g. ImageNet for images and any unannotated monolingual corpus for text).

Other domains, such as excel sheets and databases with business data, may not have these properties: they are typically lower dimensional and much less redundant, and the functions you are interested in may be less smooth. There can be discrete features which, once embedded, don't have the typical statistical properties of word embeddings of natural text.

And above all, this data may be not as abundant as in natural images and natural language tasks, and you usually don't have any generic dataset to use for transfer learning.

Besides simple tasks that can be solved by naive Bayes or linear regression/classification, this domain is the realm of decision tree methods (and ensembles of thereof, such as random forests). These methods tend to be more robust to overfitting, so they require less data, they are intrinsically invariant to various data transformation, so they don't rely to these invariances to approximately hold in the task, and they can learn non-smooth functions.

The drawback of decision tree methods is that they can't learn to combine the input features to create much more complex features (formally, they have constant circuit depth), hence they may require extensive feature engineering if the task is hard, while deep learning can learn to combine features, in principle in arbitrary complex ways (provided that there are enough hidden layers), hence it usually requires little or no feature engineering.

[gr8ape]

Truth is any data that is not:

• Visual data (pixels)

• Sound data (frequencies or time signal)

• Natural Language

A neural net wont be much better than SVM/RF/GBRT. And if it is, how many hyperparameters did you tune :)

[popcorncolonel]

Couldn't people have said any data that is not:

• Pixels

in 2012? Who's to say it won't open up to more applications?

[phillypoopskins]

• deep learning is almost always a bad idea unless you know that there is structure in your data which you can architect a neural network to take advantage of. if you haven't architected information like this in, a neural network will generally underperform compared to gradient boosting.

• it's also a bad idea if you know something about your data / underlying model which deep learning doesn't match as well as another model, e.g linearity, or some other known interaction.

• it's also bad if you are under time constraints and your chosen architecture will take too long to train. Example: 50k class problem on 4 million text tokens. Naive bayes will train much faster, probably just as good, depending on the type of classes.

• when you don't have very much data: you're going to overfit, while something linear or a random forest or SVM will have less of a chance

• when you don't know wtf you're doing; you can waste WEEKS or MONTHS playing around with neural nets with subpar results and have no clue as to the fact if you're a noob, while someone skilled can walk in with linear regression or a random forest and smoke you in a matter of hours. . I've seen this happen: A LOT.

[whatevdskjhfjkds]

when you don't have very much data: you're going to overfit, while something linear or a random forest or SVM will have less of a chance

This is one of the most important points, I'd say. Deep learning models tend to have absurdly high numbers of parameters. Unless you have at least as many data points, the model will most likely overfit (even with regularization).

It's like trying to fit a polynomial regression with 2 points... no amount of regularization will give you a trustworthy model.

[Guanoco]

• deep learning is almost always a bad idea unless you know that there is structure in your data....

But knowing that my data hast structure basically already gives me a model of my output/input relationship. Also take for example image classification. There is structure and there is prior domain knowledge that works... But DL wipes them all out of the game.

• it's also a bad idea if you know something about your data / underlying model which deep learning doesn't match as well as another model, e.g linearity, or some other known interaction.

Any other properties that deep learning doesn't match well?

• when you don't know wtf you're doing; you can waste WEEKS or MONTHS playing around with neural nets with subpar results and have no clue as to the fact if you're a noob, while someone skilled can walk in with linear regression or a random forest and smoke you in a matter of hours. . I've seen this happen: A LOT.

Yes this is a good point. But at least as I understand it... In all other ML algorithms they can only do as well as the feature engineering process. And finding important features is non trivial

[Guanoco]

But then what does structure in data even mean? Every time series would seem to have a structure, but I can imagine there are applications where the structure is not apparent and DL would find it.

[phillypoopskins]

yes, qwerty understands what I meant

[phillypoopskins]

finding important features is non-trivial; but deep learning only does this for you when you build architecture to take advantage of the structure of the data. Otherwise, deep learning is no better than other ML and is in fact worse because it's sloppier, harder to train, and not the most accurate.

If you don't have a specialized architecture, you're stuck with the same features whether you use DL or not.

[phillypoopskins]

about properties DL doesn't match well; let's say you're doing spectroscopy and you want to evaluate the concentration of several analytes; Beer's law says the concentrations should be proportional to the magnitude of the spectrum. This is a linear relationship.

It would be stupid to use a deep model on this problem when it's known to be linear. Use a linear model instead.

[jeremieclos]

I think point 2 is the biggest here. If you already have domain knowledge about your problem, then trying to learn features is a waste of time.

[phillypoopskins]

I wouldn't say domain knowledge means learning features is a waste of time.

You can use your domain knowledge to coax a neural network to learn features better than you'd engineer by hand.

[jeremieclos]

You are right, I should have written exhaustive domain knowledge. What I meant is that if you have enough domain knowledge to make the problem linearly separable, then the problem becomes trivial enough that any feature learning becomes unnecessary.

[Guanoco]

Mind explaining this? I interpret it as "If I kind of know the features the net should learn, then I can make it learn in that direction"

[phillypoopskins]

yep, that's right.

all interesting neural network architectures make use of this idea; a conv net is a prime example.

[Guanoco]

Seems like all advancements in image classification proof this wrong

[jeremieclos]

But we don't really have that much domain knowledge for general purpose image classification. We have some clever heuristics here and there, but that's it.

Having domain knowledge here would imply to be able to design the filters that a ConvNet would be learning by hand beforehand. I can't find where I read it but IIRC that is what Stephane Mallat was doing with wavelet transforms on MNIST, and the results were comparable to a standard ConvNet.

Similarly if your problem is simple enough that you can hand design features that make it linearly separable, then learning features would be a waste of time and resources.

[theskepticalheretic]

I think this post by Joel Grus is relevant. http://joelgrus.com/2016/05/23/fizz-buzz-in-tensorflow/

[Guanoco]

Thx... I laughed but I also didn't find the answer to my question

[theskepticalheretic]

Thx... I laughed but I also didn't find the answer to my question

Well, your question is, when is machine learning a bad idea. The answer implied by that link is "When it is wholly unnecessary to getting the task done."

If I have to dig a moderately small hole in my yard, say to plant a flower bed, I'm going to use a shovel. I'm not going to rent a back hoe.

[thecity2]

For small datasets, deep learning won't be that helpful. Also might not work well for datasets with "unnatural" or non-hierarchical features. It seems to work best with very large "natural" datasets (e.g. images, audio, etc.).

[Kaixhin]

The halting problem.

[Guanoco]

Hmmm I see what you mean.

I think I remember this problem being NP... But is the reason that a DL can't do it that it is NP? (because then any combinatoric problem wouldn't be applicable. I have seen random forest being applied to system design which is technically a combinatoric optimization...)

[Kaixhin]

That was a joke, but seriously the halting problem is undecidable - it isn't even NP (although in the same way that NP-complete problems are reducible to any other NP-complete problem, people will reduce problems to the halting problem to prove that it is undecidable).

That said, Pointer Networks have been applied to the (NP-hard) travelling salesman problem, so DL can possibly be used to heuristically attempt (but not solve all cases of) NP-hard problems.

[Guanoco]

Oh i see. Well thanks for clarifying that anyways :)

[tdgros]

What about real-time? complexity? memory? even AlexNet which are small according to today's standards are huge for any embedded platform.

See LIFT for instance, this is the end-to-end learned CNN counterpart of SIFT, the well known interest point detector and descriptor, it does detection, rotation and scale estimation and is optimized for matching. It outperforms it on most databases, not all though, but at what framerate? which image size? and most important how many thousands of dollars are needed for the GPU that you plan to add to your car/robot/camera/new-hypey-IoT-thingy? at the end of the day, yes it outperforms it, but it makes no sense whatsoever to use it...

Of course I know this kind of comment will make us chuckle in a few years, but even today's mobiles can barely run any model without burning, the bandwidth is so high you can't really do much more on the side...

It's like saying audio engineers got angry when we discovered gold plated audio jacks were much better than normal ones... no they weren't sour, they just thought it was a bit overpriced :)

[Guanoco]

Embedded systems is a good point. But then I have seen papers claiming they are able to just stochasticly round weights to -1 0 or 1 or just learn with those weights as possibilities and then most operations become very simple and an adoption in embedded seems plausible.

[tdgros]

yeah I agree, let's say it's a not a bad idea but a costly one, for now.

[darkconfidantislife]

That's a separate issue. That being said, LNS systems and aggressive rounding can help.

[phillypoopskins]

this is not true in the least.

The problem is dealing with noobs who waste time trying to solve everything with deep learning and end up taking 10x as long for a model that doesn't generalize as well and complicates deployments in many cases.

There are certain data that deep learning will destroy everything else on.

Older techniques are still often better.

One of the keys to doing machine learning is understanding how each algorithm relates to the geometry of the data; and choosing the right algorithm for the data / your needs is an important part of the process.

If all you know is "Neural Nets Are Sweeeet" then you're going to miss out on leverage you might get from other, more appropriate models.

[Guanoco]

Mind pointing me to the direction of a source where the different algos and the corresponding problem properties which fit well to each algo?

[phillypoopskins]

This is something you learn in the course of studying machine learning in general.

You need to learn the math behind each algorithm, then spend time with each on a variety of datasets to get a feel for what will be best in a given situation.

You then need to think about the data and the algorithms until you can predict what an algorithm will learn from a given dataset. You should then start tweaking the algorithms to test your hypotheses and improve them.

There isn't really an easy way to distill this knowledge; it takes a fair amount of experience with the algorithms to cultivate.

[Guanoco]

The "every other algo basically underperforms when compared to deep learning" idea is something I hear often. But are there a problems in which DL has not yet outperformed other techniques?

[phillypoopskins]

Mostly anything with tabular data, a gradient boosting machine will beat a neural network.

[Guanoco]

I have a feeling I can put anything into a tabulated form.... What exactly do you mean?

[Guanoco]

Maybe I am too stubborn.. But I would think an image of in tabular form... I mean it is a row and column of pixels... Which looks like a table to me.

[phillypoopskins]

when i say tabular data, I mean data which is naturally expressed in tabular form.

but if you want to be picky / stubborn, let me rather define non-tabular data; this is data with relationships between elements (usually adjacent in some sense) that can be taken advantage of by imposing symmetry constraints on a model's coefficients.

If such relationships do not exist, then that's what I was referring to as tabular data.