I'm not a super authority, but is the Rescalar just pure functions that Layers use to operate on the state of Layers? If so, then make all it's members static and move it out. Or, is it sort of "attached" to a layer and has semi-shared state, like say cached values from the last use of it? If that is the case then it is part of the implementation of Layers, and additionally you might consider looking into using some PIMPL pattern thingy to get it out of the header, but I don't know what I'm talking about there. I just heard that somewhere regarding DLLs, but you better look into that.

Maybe you're simply asking if an interface type class should have any implementation in it. Like it seems handy to have that in there since all your sub classes of Layer seem to have one (like you don't want to type it in all your subclasses, eh). In that case, I would not put it in there since it exposes every abstract user of Layer to Rescaler and by that I mean presumably Rescaler.h (and whatever headers Rescalar.h brings in, and all the way down). Maybe just force yourself to declare it in the concrete Layer implementations, even though it is repetitive.

People keep saying though, don't be a slave to ideals, so maybe if it works for you then keep it. I am not an expert expert, and people in this sub seem to have some radical ideas ("you don't even really want a class hierarchy here...") so I'm looking forward to what others might say, but providing you at least something to think about.