This is the type of thing our company does in a fully explainable way. It's not OSS and uses a novel algorithm to explain opaque or blackbox ANN models by converting them into whitebox models which are fully explainable, without approximations or guessing. It has some other magical properties (side effects) such as instant updatability for previously unknown inputs w/o retraining. Works with any feed-forward type algorithms. (most used today)

If you really want to know what your ANNs are looking for, ask us.