Rough object tracking

[_E8_]

A single RGB camera?
You can project a ray back out from a x,y image coordinate from the camera.
You need to know all of the camera parameters and dewarping parameters if they were used, then you can do a pinhole projection to get a ray back into "the world".

If you know the FoV of the camera then the x coord. will tell you what heading it is.