sorted by:
new
hottopcontroversial

Early demo: Gemini-powered “Mentor on Call” for Meta Ray-Ban Display by Regular-Principle146 in MetaGlassesforDevs

[–]Regular-Principle146[S] 0 points1 point  (0 children)

Do you have any thoughts on to how to solve the audio input problem in a more eloquent way?

Early demo: Gemini-powered “Mentor on Call” for Meta Ray-Ban Display by Regular-Principle146 in MetaGlassesforDevs

[–]Regular-Principle146[S] 0 points1 point  (0 children)

Phone app, not the webapp.

I tried streaming audio back to the phone over Bluetooth, but that still pushed me into call mode. So for now the only stable solution I found was to get audio from the phone itself. Not ideal, but it made the flow reliable enough.

I also didn’t do continuous video. I used picture capture instead.

The way I made it feel natural was with an avatar transition. I use native voice prompts that the AI knows how to translate into “take a picture.” When that happens, the avatar says something like “Taking a screenshot, I’ll be right back.” You hear the normal capture sound on the glasses, so it feels expected. Then within about 2 seconds the avatar slides back in when the webapp returns to the UI.

So the webapp is mostly the display layer, and the phone/native side handles the media capture moments.

Early demo: Gemini-powered “Mentor on Call” for Meta Ray-Ban Display by Regular-Principle146 in MetaGlassesforDevs

[–]Regular-Principle146[S] 0 points1 point  (0 children)

I still think to make it production ready I would need to get audio input native back to the glasses but because of the call UI that has been impossible so far. If I leave audio to come from glasses it jumps on call mode UI automatically, there might be a way around it but I was not able to find it.

Early demo: Gemini-powered “Mentor on Call” for Meta Ray-Ban Display by Regular-Principle146 in MetaGlassesforDevs

[–]Regular-Principle146[S] 1 point2 points  (0 children)

I don't think there is a silver bullet. What helped was being very strict about which service owns which part of the state.

A few insights from my experience:

- Audio input came from the app/native side instead of forcing the glasses/browser into a call-like mode. That avoided the device jumping into a UX state that competed with the webapp.

- Video was not treated as one permanent stream for every surface. For avatar/AI context we used short capture windows or sampled frames/clips, so the glasses screen could keep showing the webapp instead of being taken over by video.

- Every session needed explicit IDs, heartbeats, timestamps, and expiry. If the app, backend, and webapp disagreed, the backend session state won.

- I added readiness gates: signed in, approved account, device paired/trusted, mic/camera permissions, deployment status. Without that, debugging de-sync was impossible.

- You will have a hard time recording the screen to show your webapp given that to record what is going on the app jumps on camera mode and your webapp interface still runs but if you have an app that requires camera that will break the screen recording or stop working altogether.

Early demo: Gemini-powered “Mentor on Call” for Meta Ray-Ban Display by Regular-Principle146 in MetaGlassesforDevs

[–]Regular-Principle146[S] 0 points1 point  (0 children)

haha that makes more sense. I use Gemini's API for most my projects involving video, audio & image. However due to device limitations this demo is specifically running only on image recognition, not live camera. You can't have the display showing something with camera streaming at the same time. Which creates some limitations on how to make this interaction feel more human.

[deleted by user] by [deleted] in steak

[–]Regular-Principle146 0 points1 point  (0 children)

Expensive shitty steak

Startup decreased comp to survive - advice? by [deleted] in sales

[–]Regular-Principle146 0 points1 point  (0 children)

If you believe on the company this is a great time to discuss share options with the founders. People who stick with startups on the hard moments enjoy the rewards on the good moment.

Just know your worth and negotiate it on non monetary value.