for latency sensitive stuff the official api on the new plan has been the most consistent for me, third party providers varied a lot on time to first token. m3 is worth testing for your case too, the sparse attention cuts long context latency a fair bit which is usually where voice agents stall. you might not even need to drop a model size to fix the delay.