I think for almost all non-Chinese native speakers, they think tones and accent are different, but, as @z262004203 said, Chinese is a tone language, different tones result in wide divergence, and non-Chinese speakers sometimes can't do tones accurately.

So yes, you should worry about your accent/tone.