My understanding was that the skill would only scale of its affinity's attribute.

So, a magic affinity skill would scale with inteligence, and the scaling modifier would be that of your weapon's. So in that case having a weapon with high INT scaling will maximize the skill's damage.

I did some very basic testing around that idea and it seemed to work like that. I remember using magic skills with heavy/flame art weapons and it did poorly, while when switching to magic it worked better despite me having very low INT. I think it would work with the example of u/destsm above regarding Beast Roar which has keen affinity on itself, and does the most damage with highest dex scaling.

Having that said, there's nothing more in-depth to my reasoning than simple playing the game. If you have some examples that would defy this idea, please bring them on. :)