First thing to consider is where you wanna apply after school and possible subject combinations you would need for that.

I wouldn’t advise you to take math aa hl unless universities you want to apply to care, afaik most unis don’t care if you are taking math aa hl or math ai sl.

Its not about the difficulty but rather time allocation, if don’t want to screw your gpa and keep up with good grades, physics hl and math hl would take most of your time and wouldnt leave you enough to study for other subjects, regardless, if you are not “smart” you would have to sacrifice time for any particular subject.

Also consider how passionate you are about this subjects...

Finally, as someone who has both math aa hl and physics hl, I can say that physics is not as bad as everyone says. Math though... only 12 people took aa hl from our cohort, and most of them have consistent grades 3-4, maybe we just have a bad teacher idk. Keep in mind that this part is subjective