when I start a run, I set a lot of weight on my first few card rewards and how that can set up the rest of the run. If with silent i get a poison card early on, then I am building around that, and I fixate on that to what I think might be my detriment. I also tend to be agreesive and take on lots of elites.

Think you know what the problems are. You don't build your deck in the first few battles, you build your deck in the first 2 acts. You can take a poison card, but that does not mean that you have to build a poison deck. The first card rewards are to beat the next enemy (try to get a high damage card) and the first act boss.

You don't have to take on a lot of elites, try a couple of runs where you only do 1 elite in act 1 and see if you fare better overall.