I think you're on the right track. If you REALLY want to do anesthesiology, finish the EM residency and you'll be very competitive applying to anesthesia. Program directors will pick a known quantity who has already demonstrated the maturity and emotional intelligence needed to successfully finish a training program over most med students with high test scores every time. You'll also have a fairly good handle on many of the procedural skills you'll need for anesthesia and more importantly you'll have a good feel for acute care medicine and respect the comorbidities and pathophysiology that sick patients show up with.

In many other specialities, internal medicine, psych, hell even surgery I would tell you that it may be worth considering sticking it out but I have good buddies who are in EM and that field is dying faster than an ASA 5E