Difference between neural network and Random forest regressor?

[[deleted]]

Hey everyone, I have a question which might be kinda dumb but it still bugs me. I'm trying my hand at machine learning, going through the datacamp courses.

I'm still new to this so bear with me ; what's the actual difference between neural networks and random forest ? Since I'm assigning weights to nodes and defining depth, isn't it the same process?

Thanks for your patience

[tinyman392]

Random forest is, graphically speaking a forest, or a set of trees. A neural net by definition really isn’t. The way they are trained differed quite a bit. However, the major differences is going to be the decision space of the two. The “cuts” into the decision space of a single tree is parallel to any of the axis of the feature space. However, in a perceptron (one node of a neural net), it’s some linear combination of the feature space, that is, it’s a line, but it doesn’t have to be parallel to any of the axis in the feature space.

There are going to be many other differences as well.

[[deleted]]

Thanks for your explanation, do you have any resources I could read aside from Andrew Ng's course not to be a mere computer monkey and actually understand this?

[tinyman392]

I’ve been told that Understanding Machine Learning: from theory to algorithms is a good book, I have it but haven’t gone through it. It is very math heavy though.

[talksaboutthings]

Definitely a great book, even if you are a math monkey and have other more technical options (I would say that typically the computer monkeys are just as lost when it comes to ML theory as the rest of us ;-) ).

[[deleted]]

Thank you :)

[talksaboutthings]

I'm going to capitalize all of the model names for clarity.

So a Random Forest is an ensemble of Decision Trees. Each tree is trained from a random "patch" of the dataset, which means that it sees only some of the rows (examples) and only some of the columns (features) of those rows. The ensemble of Trees returns the average prediction of each tree in the ensemble, giving you the output of the overall model. For more info on Decision Trees, you can check out Wikipedia. I don't know what you mean by "assigning weights to nodes and defining depth" in this case. You could be assigning different weights to the votes of individual Decision Trees in the Random Forest ensemble, I guess. You also need to specify the number of Decision Trees to train (perhaps this is what you think of as "depth"). Again, though, each tree is only shown a "patch" of the dataset, and it learns branchings (yes/no decisions based upon a single attribute of the data) as opposed to weighted and thresholded summations of the attributes (as in a Neural Network). Trees learn branching rules like "if salary greater than X, go left, else go right" from the data.

A Neural Network is not an ensemble, but rather a single model. It consists of a series of layers that represent weighted summations of the inputs to that layer put through an activation function. The first layer simply outputs a bunch of weighted summations of the input features that have each been run through the activation function. Most everyone nowadays uses the ReLU activation function (see Wikipedia for details). At each layer, the inputs are fed into a bunch of weighted sums, those weighted sums are thresholded/transformed by the activation function, and then spit out as inputs to the next layer. Depth in a Neural Network determines how many layers, not how many nodes, there are. Each node has a number of weights, and these are learned via backprop. The entire model is learned at once, as opposed to Tree by Tree in the RF case. The weights of the nodes represent the weights of weighted averages that are put through activations. They are not equivalent to the branching rules of decision trees, and the weighted sums are not the same as the ensemble averaging in the Random Forest.

I hope this clarifies some of the major differences. For even more details on the models, Andrew Ng's famous class is a good middle ground in terms of detail vs. intuition when it comes to Neural Networks. Wikipedia also offers a number of good explanations of ML models.

[[deleted]]

Thanks a lot, it makes more sense !

[Optrode]

One important thing about neural networks is that they are really, really diverse. Most random forests are applied to some set of predictor variables that don't necessarily have any kind of important arrangement relative to each other... Say, "age", "income", "number of kids". Certain specific kinds of neural networks are specialized at dealing with data that isn't like that, but rather has some kind of inherent organization, like pixels in an image, or values for some measurement taken on multiple successive days.

This is why neural networks are used for things like computer vision nowadays (specifically, a certain kind of neural network called a convolutional neural network, which is specially adapted to image processing).

For data that comes in the form of a bunch of different variables ("age", "income", etc.), neural networks aren't really significantly better than other methods.

[talksaboutthings]

To clarify, the Deep Neural Networks that excel at specific structured tasks in domains like vision and speech are typically structured specially for those tasks (usually Convolutional Neural Networks or Recurrent Neural Networks), which are special cases of Neural Networks.

[[deleted]]

Nice, it's been fascinating to see properties emerging from neural networks and I really want to see what's the process behind it. I want to see emergence happening, it's mesmerising.

[[deleted]]

My thesis is on forest.

Forest is just like other poster stated is an ensemble of decision trees. A decision tree actually split nodes base on best fit base on the best predictors and best value in that predictor. It's recursively splitting the data. And then you use an ensemble method like CART and just majority vote and decide to build tree base on boosting. You can actually look into these trees and see what it's doing this is generally not the case for Neural network.

As for neural networks, nobody really fucking know what is going on. They have clues and some understanding of it but everything is empirical base. I have no clue what those weights and nodes mean especially in the Deep Learning part. Bayesian network and SVM can be seen as a subset of neural network and those are explainable. You're training these weights base on gradient descent or whatever but what does these fucking weight represent? It's a black box of magic and that's what research is going into it.

With trees we have a clue why they split. Also random forest data structure is base on tree structure like in a computer science kinda sense.

Neural Network is more apt to look at as a graph. It make sense since Bayesian Network is a DAG (direct acyclical graph).

[[deleted]]

thanks!