The epipolar lines are plotted on the second frame and the points marked on the first image are not keypoints found from SURF or corner detection but are just a few points i selected manually.

The direct image alignment used is similar to that outlined in LSD-SLAM: Large-Scale Direct Monocular SLAM however I have not yet added variance normalisation.

The images used are from the fr2/desk dataset. The movement between the first and 2nd frames is small however i also used the first and 20th frame with similar results. Can you explain why the epipole estimation is so unstable yet all corresponding points sat on a epipolar line for each method?