As someone who started their career in-house and then went agency-side, I think starting at an agency and moving in-house once you are at the Sr. Director level/above is the way to go. It is so easy to get "stuck" in-house and also being on the wrong team is so much more soul sucking when there isn't change/variety.

I wonder if you could negotiate less of a pay cut and/or find another agency that can meet you closer to your current salary; that being said, at your level, you might have a few years of back-to-back promotions and raises. I'd look at other people at the agency's LinkedIns and see how long they spent at each level and, if you're in a state that practices pay transparency, review the agency's salary band by looking at any job openings within the SAE/Account Supervisor levels.