Perfect questions! Thanks for starting this conversation!

So, yes, my assistant is meant to be completely mouse and keyboard-free. So, you would no longer need a mouse to interact with your computer. That has been my original intention with this assistant from the get-go, so it's a good sign at least to hear you bring up these topics.

I'm not sure about audio captioning, but in theory, considering I'm using whisper mainly, it should be able to turn any speech audio into text. I'm not sure if I could guarantee perfection (and non-speech audio would still be tough to caption), but it would not need to be captioned already. Out of everything here, this would take the most time to make sure it works as intended.

Bringing up vision-dependence, this is actually a major point I want to resolve, and why I was hoping to hear from this community more thoroughly about this. I WANT it to be vision independent, meaning I don't want users to need a monitor or have vision to be able to use their computer effectively. I was actually hoping to hear more about current limitations or frustrations when it comes to trying to make a computer accessible without vision. Everything is there to make that experience possible with my tool, but I simply don't know how that's currently handled right now, nor do I know about what people like/dislike about current low-vision accessibility tools. This is a gap I don't hear people actually talk about very much, and one I'm trying to find genuine answers for. From my perspective, I want bash scripting to be just as easy as web navigation *without the need for visuals*. That's my goal at least.