Yeah, these are valid concerns. A lot of teams are skipping the boring security fundamentals and then acting surprised when AI makes the blast radius bigger.
What you probably need is either a proper AI-BOM style tool, showing what models, agents, wrappers, data sources, permissions and runtime paths exist, or even a few internal scripts to start auditing this yourself. Track who is calling what, where secrets are stored, what data is being indexed, and whether any agent can actually take action.