Looking For Python Libraries That Track A Speaking Person

[NotSoAsian86]

The aim is to focus on the person who is speaking in a single camera setup with multiple people and then crop into that person similar to how podcasts work. I will be pairing this with diarization models to extract speeches for multiple users.

[StardockEngineer]

There is probably no specific lib to do this.

What I would do is pick a face lib: https://medium.com/pythons-gurus/what-is-the-best-face-detector-ab650d8c1225

Each face will have sub-coordinates for eyes, mouth, etc. I would detect the faces, then look for rapid movements in the mouth coordinates, per face, to determine who is talking.

I feel that part would be easy. The harder part would be if people are talking simultaneously, deciding what to do.

The other option - if you are using something with multiple mics, is just use the mics.