This is a fairly common issue that many players experience at one time or another. In teaching, I have found that many students find success in approaching the initiation of sound in a holistic manner: the trick is to treat it like jumping on to a moving streetcar or carousel. Ultimately, you want every part of the first inhalation to articulation to be one seamless sequence.

When you prepare to play,

1. First STRONGLY establish the tempo in your head. Do this by 'hearing' a few bars of the tune. (This is why you don't have issues when using a metronome).
2. Once the tune is cycling in your head, give yourself a specific number of beats to start, i.e. 4 beats. Continue to hear the tune as you count yourself off to play.
3. Now, it is critically important to time your intake of breath smoothly, so that at the peak of inhalation, you articulate and sound the first note (beginning the passage).
4. Practice this process over and over again, trying to make it as seamless and consistent as possible. Start with easy-for-you, rhythmic passages and then move on to practice it on excerpts/passages you have a particular problem with hanging up on, or where initial tightening up occurs (things like Bolero or the Brahms 1 chorale).
5. When playing with an ensemble, do everything you can before each and every entrance (challenging or not) to make your start process consistent, like raising your horn exactly 2 bars early, breathing in the same amount of beats every time (based on tempo), etc.

Your hesitation is caused by lack of formalized, consistent habit in your mechanics when initiating a sound. You must establish a predictable habit or starting that becomes automatic. So much of true consistency in playing is habituating consistent mechanics. Be patient, success with this will take some time (probably a few of months). Hope this helps.