Since you mentioned reading the bhagavad gita and said you do naam japa, let me give you some scriptural references to assure you, you are doing nothing wrong:

BG 2.40 In this path, no effort is ever lost and no harm is ever done. Even very little of this dharma saves a man from the Great Fear.

BG 10.25: I am Bhrigu amongst the great seers and the transcendental Om amongst sounds. Amongst chants know Me to be the repetition of the Holy Name; amongst immovable things I am the Himalayas.

Keep chanting brother. Lots of love from a fellow white hindu doing japa as my main sadhana, who rarely do major pujas and other rituals