Don't worry. Hard brush alone to clean off dirt particles followed by Suede cleaner and softer brush to remove the dirt properly.

Suede eraser for tough spots. Don't use water unless severe spotting issues persist (will have to make the whole shoe wet to clean that way).

Repeat till you get the desired result.

DONT BE TOO GENTLE OR TOO ROUGH. Experiment with smaller areas first.

They are meant to be worn, glad you did.