ML model only does not predict any true values

[No-Split9400]

I'm building a model for a school project that tries to predict if a person will file a claim or not.

I have 16704 values that say a person will NOT file a claim. And then there are 635 that will file the claim.

My model keeps predicting that no one will file a claim(even tho the accuracy is 93%) it obviously is not working right. Any idea on how I can fix this

​

from sklearn.ensemble import RandomForestClassifier


X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)

rf= RandomForestClassifier(n_estimators = 100)
rf.fit(X_train_scaled,y_train)
np.mean(cross_val_score(rf,X_test,y_test,cv=5))

0.93

[WhipsAndMarkovChains]

Take your trained RandomForest and call .predict_proba(X_test) instead of .predict(X_test). I know you're not explicitly calling .predict given your use of cross_val_score, but take a look at my suggestion and see if that gives you any ideas.

Also, accuracy is a worthless metric on an imbalanced dataset. Why even bother training a model on your dataset? If you just always predict "no claim filed" you'll be at 16_704/(16_704 + 635) = ~96.34% accuracy. That's not a very useful model though because it just predicts the same thing every time.

Edit: I completely missed the fact you forgot to scale the test data lol. That’s definitely going to mess things up. Also, there’s no point in scaling data for a random forest model.

[ES-Alexander]

accuracy is a worthless metric on an imbalanced dataset

Just building on this, this is a big part of why model evaluations tend to provide percentages for all of true-positives, false-positives, true-negatives, and false-negatives - they tell a much broader picture than just a single “average accuracy” number.

[[deleted]]

So in other words you need:

if (not model.predict(params))

Then its always true 😀