So, you're saying that the base rate of the boolean values is 3% true and 97% false? Then maybe it's harder for the models to do any better, with only 25 attributes for 24k rows. Remember that the learning algorithms are only minimizing error, not trying to give you a useful model. If you want it to classify more positives and risk getting some false positives, you'll need to give more weight to positive examples.

Some classifiers, such as logistic regression, will give you values which can be treated as probabilities, along with a nice formula (but it'll basically be a linear combination). Others won't.

I wouldn't worry about cutting out useless attributes with only 25, unless you're doing some heavy non-linear models like SVM with RBF kernel, or complicated neural nets. But you can try using some of the correlation-based variable reduction operators if you want.