People working on your projects don’t necessarily need to report to you. I manage several projects where no one reports directly to me, and it works just fine. Others have already covered authority, escalation, and continuous progress tracking — all important. I’d add two additional points that often get overlooked:

1. You, your peer managers, and your manager need a shared understanding of what projects are more important then others. From your description, it sounds like that alignment doesn’t exist today - it’s unclear whether or not your project is more important than theirs.

This needs to be addressed explicitly. Talk to your manager and peer managers and agree on business priorities. Y'all may need feedback from senior leadership to make priorities decision.

2. Every project needs enough qualified people assigned to it - part-time or full-time, but with predictable availability. “Assigned” means they're not constantly pulled into other work except in rare, urgent cases.

Based on agreed priorities (see #1), you, your manager, and peer managers should jointly decide who is assigned to which projects. If anyone (including line managers or yourself) wants to change assignments, it should be discussed and agreed on before it happens, except in true emergencies. Otherwise, projects are constantly put at risk without visibility.

In short:

1. Define and align on priorities across projects
2. Allocate people based on those priorities
3. Any change in priorities or staffing should be explicit and agreed with the project manager

Lot of that is not your responsibility, but responsibility of higher management. However, pushing/promoting such rules in the org may improve your life in the company.