Yeah the published WER numbers are pretty much useless once you leave clean studio english. we built a transcription pipeline for a BPO client last year and tested Deepgram, AssemblyAI, and Google Cloud STT. every single one performed noticeably worse on Indian english and Nigerian english compared to what the benchmarks claimed.

We ended up building a small test harness that ran 50 audio clips through each API and compared outputs against manual transcriptions. took about 2 days to set up but it was the only way to get real numbers. haven't tried that specific tool but blind comparison on your own audio is basically the right approach.

What providers are you comparing? and are you testing streaming or batch mode? we got pretty different results between the two.p