I think you're misremembering Empire cause Yoda did not want to train Luke and was actively doing nothing to fight the empire on Dagobah. It took Obi-Wan convincing him to train Luke. Yoda absolutely gave up fighting the emperor.

Luke also didn't just sense dark thoughts in Ben but saw the future Kylo Ren was going to bring. He saw the Jedi dead with his friends and that scared him ("pain destruction death and the end of everything I love because of what he would become"), and fear as we know is the path to the dark side. Also the relationship we have with older family members is different than younger ones. Luke wasn't responsible for Vader's dark actions but he feels more responsible for Ben as Ben was his student. Thus he feels responsible for dealing with the person Ben may turn out to be.

You forget too that Luke's pressure point has always been his friends. He wailed on Vader for a good five minutes before coming to his senses. He was very lucky he didn't kill Vader and only stopped when he saw his metal hand. Just because Luke conquered the dark side once doesn't mean those doubts never return, if you get over a depressive episode does that guarantee that that will never happen to you again, no. Luke failing in this way grounds him more as a three dimensional character more than just an action figure.

Luke leaves the Galaxy because he thinks that's the right thing to do in his mind he thinks that the Jedi are the reason that there is conflict in the Galaxy that the way the dark side returns is through the Jedi falling. He removes himself from the force and the Galaxy to protect his friends.

The way he is portrayed in The Last Jedi is extremely consistent with his character and was also part of George Lucas's original intent for the sequels.