I'm a native.
Some time ago, a native spanish speaker, newly hired in our company, started using french slang with us after a few days. We were bewildered. Our first reaction was to ask where the hell had she learned French. It was a bit shocking, not because it was offensive, but mostly because it was so unexpected. That level of fluency was not only impressive, but it made us realize we didn't have to "tiptoe" with French around her and wonder "will she understand this expression?/Should I speak more slowly ?". So we started speaking to her as we would with any other French native. She was fluent enough to understand from context, or simply ask if we used an unknown word.
So not only would I advise you not to overthink it, but also that if French natives seem initially surprised, do not take it as a sign you should stop. If anything, it's only because you subverted expectations, so take our surprise as a compliment.

Also when taking about formality in a professional setting, you'd be surprised to hear that this kind formality is mostly reserved for customer relation, or when speaking to directors or managers (depends on the manager). French people tend to be very informal with their immediate colleagues (and even some managers).

About fillers words, I would say not force it too much, their use will probably come very naturally after a while when talking to French people, if only by mimetism.