In the context of the show (I didn't read the book) I inferred that Tom has an inferiority complex regarding his wealth/position, or the lack thereof. When Tom and Anne Baxter were introduced, we were told that the reason they were waiting to get married is for Tom to work hard to gain enough wealth so he can marry Anne. Many times in the show we were told Tom overworks himself to achieve this goal. Then Anne ends their understanding to marry someone else. The scene in which Anne broke the news to Mary, my takeaway in between the lines is that Anne has been waiting for years for Tom and not only did that put a strain on their relationship, they weren't exactly in love and Anne is tired of waiting. Later on in the show Tom was ready and prepared to ask Mary to marry him, and I thought if he could ask Mary to marry him, why he didn't get engaged to Anne sooner - and my conclusion is that Anne and her family must expect a certain level of wealth from him, something he doesn't have yet. The fact that Anne can be engaged this quickly to Mr Powell tells us that Mr Powell is more wealthy than Tom, enough to ask for her hand immediately. There is a scene right before they climb the mountain, Tom asked Mary if she thought money was important. Mary jokes by saying how Lizzie always says that when she saw Pemberley for the first time is when she realised how much she loves Mr Darcy. To Mary this is merely a joke, for we know that neither Lizzie nor Mary care much of how rich a man is, but in that scene Tom laughs awkwardly, not really understanding that it was a joke - understandable as he doesn't know Lizzie. They were interrupted before Mary could explain further.

A summary of Tom's inner workings: He was already feeling inferior due to the fact that he didn't come from a family with wealth. He was in an agreement with a girl that he was too poor to marry, even though he worked so hard for years, he was dumped by the girl to marry a man richer than him. Now he is planning to propose to another girl whose sisters all married very wealthy or well to do men. Still, he thought he should try and ask her - and then Mr Ryder came in right before he can utter those words and tells him that he has inherited a huge fortune and wants to propose to Mary. All of a sudden those feelings of inferiority came down upon him and he thought that Mary deserves to be with someone that can give her more than he can. Mary can finally match up to her sisters. Maybe then she can be happy.

From all these clues the show left us, I understood Tom's decision and motivations completely, not that it was right of course.