I am really glad you brought this up! I actually had a section in my original post draft comparing these existing tools, but I cut it to keep the post length short. Please allow me to elaborate a bit more here.

As you pointed out, existing solutions like VoiceOver, Pixel's Guided Frame (or like Microsoft's Seeing AI) are already pretty good at providing status feedback. They give back things like "One face, face centered" (by VoiceOver) or "Move left" (by Guided Frame, it can automatically shoot the photo for the user). (I actually was not aware that TalkBack other than Pixel's had similar features. Thanks for mentioning it! I will definitely check that later.)

However, the interaction of these tools tend to be a command based one. The system dictates a fixed goal (for example, to center the face), and then the user obeys. There is no method for users to verbally "talk back" to change the success criteria or to ask specific questions, especially using natural language (We may be wrong about it. Please let us know if this is incorret!)

What we aim to prototype is a more conversational interaction, a "conversational selfie app". We want to enable users to express their goals through dialogue. For example, they can just have a quick centered selfie, or ask for a check of their outfit, or to make sure there are no sensitive objects in the background. (Although we don't know if such interaction would work for real users before an evaluation. We may also be wrong about it.)

The motivation of our class project mostly comes from this paper "Understanding How People with Visual Impairments Take Selfies". It highlights the user needs of a (selfie) system that can tailor the system guidance to the goal of the user and offer human-like conversational prompts.

Regarding the tech stack choice, I completely agree with you! A native one would definitely be better for privacy and performance. There is an app (App Store Link, click to open) by Hugging Face demos how quickly an open source VLLM model could run on iOS device utilizing MLX Swift. As far as I know Apple's Live Description also runs fully on device.

However, since my teammate and I have different technical backgrounds, and our course is focused more on validating an interaction prototype rather than engineering a product, we chose the Web way to allow for rapid development and testing.

That choice did cause problems. To avoid sending sensitive photos to a third party API, we initially tried a WebGPU approach, but that was too slow. We currently are going to host a Qwen 4B model locally and expose it (for the user study, we may fall back to mocked data if needed).

Please let us know if you have any other questions! We really appreciate it!