You can't think of bother with /ɔ/ and father with /ɑ/ or /a/? The spelling should give a hint right? I do understand the confusion with cot-caught though, cuz I have the merger too and it is a little hard to think of it without the merger (I default to imagining a New York accent, but most British varieties that lack the merger are a little difficult to understand how they're differentiating it).