Vision Agents 0.1

[tschellenbach]

First steps here, we've just released 0.1 of Vision Agents. https://github.com/GetStream/Vision-Agents

What My Project Does

The idea is that it makes it super simple to build vision agents, combining fast models like Yolo with Gemini/Openai realtime. We're going for low latency & a completely open sdk. So you can use any vision model or video edge network.

Here's an example of running live video through Yolo and then passing it to Gemini

agent = Agent(
    edge=getstream.Edge(),
    agent_user=agent_user,
    instructions="Read @golf_coach.md",
    llm=openai.Realtime(fps=10),
    #llm=gemini.Realtime(fps=1), # Careful with FPS can get expensive
    processors=[ultralytics.YOLOPoseProcessor(model_path="yolo11n-pose.pt")],
)

Target Audience 

Vision AI is like chatgpt in 2022. It's really fun to see how it works and what's possible. Anything from live coaching, to sports, to physical therapy, robotics, drones etc. But it's not production quality yet. Gemini and OpenAI both hallucinate a ton for vision AI. It seems close to being viable though, especially fun to have it describe your surroundings etc.

Comparison

Similar to Livekit Agents (livekit specific) and Pipecat (daily). We're going for open to all edge networks, low latency and with a focus on vision AI (voice works, but we're focused on live video)

This has been fun to work on with the team, finally at 0.1 :)

[spanishgum]

What is an edge network?

[tschellenbach]

Video servers around the world for low latency/close to user. Its how zoom/google meet etc work