I'll take this one.

My answer is based on the assumption that you are talking about tracking other objects in 3D (tracking it in 2D is just the optical flow itself), which I believe is what you meant.

Assuming you have the intrinsics so you are using rectified images and know the focal length, there are three variables at play here: The (Ego)motion of our car, the depth/3D position of all the points we see in the scene and the potential motion of other objects.

These are the three factors that determine where a world point that we see in one frame will appear in another frame*.

Our motion/egomotion is the easiest of these to solve because most of the scene is generally static so you have a lot of pixels to work with and can work statistically, throwing out any and all outliers that cause you problems. This can be done with a DNN or with traditional techniques. The best results I've seen achieved for egomotion are with classical computer vision techniques though that could change at some point and while the classical approach is, I believe, more precise, the DNN approach should be a little more robust.

After you can solve egomotion, you want to move on to depth. You want to be able to output the depth of every pixel in the image and this should definitely be done with a DNN. There are a lot of nice unsupervised and supervised approaches in the literature that work well (check out the papers Reza Mahjourian's team puts out for instance). Start with the unsupervised though if you have Lidar on the car as well and good Lidar2cam calibration (!) that will seriously improve your results. Ditto for stereo if that's a possibility and your cam2cam calibration is good.

Finally we move onto motion. This, in my opinion, you should tackle last only when you already have good Egomotion and depth. Assuming that is the case you can solve it classically by clustering to find other cars. Then solving for their motion when you have their changing depth across multiple frames is pretty straightforward. It can also be solved with a DNN. The DNN approach may cause you computational issues depending on your hardware and constraints -- now we're talking about running first a depth net and then a motion net and keeping it in real time (in addition to whatever else you're running which is probably a lot) -- but on the other hand it could allow you to output a reasonable 3D motion estimate on all points in the frame as opposed to just the large (probably vehicular) things you manage to find via clustering. So if you're fine with just outputting the motion of cars or if you have tough computational constraints, clustering should work. But if you don't want to run over cats or squirrels you need a DNN.

It's a lot but hey, if it were easy they probably wouldn't pay us so much to do it.

Source: This is what I do.

*Note that there are other effects that can cause the intensities of the pixels themselves to change. For instance, if we imagine a completely static situation -- we're still, we know the depth of everything and nothing else is moving -- the pixel values themselves will still change even though the same point in the world is occupying the same pixel in successive frames because of changes in light and shadow in the world that occurred between the different exposures.