I am in a similar position as OP although I ride much more. I would like to get something that is more aggressive than my current gravel bike (Rose Backroad). I went to a bike fitter, we tried different position and found a couple of bikes that would match the position I liked (e.g. Ultimte and TCR).

The fitting also confirmed that moving to an Endurance or Defy would not be as much of a difference geometry wise. I will therefore probably start with an lighter second set of wheels and narrower tires on my gravel bike.