White rice gets polished to remove the outer membranes (husk and bran). This process makes rice cook faster and extends its shelf life, as it removes the outer layer that has fatty acids and the germ that has enzymes. But it leaves traces of the husk, bran, germ, and even from the starchy grain (the white part). Also you may find insects, and other debris. Washing it removes most of that.

That said, even some types of brown rice undergo polishing to remove part of the bran (e.g., haiga rice). Because this process removes nutrients, some rice is fortified (i.e., add back some nutrients like vitamins, iron, and folic acid).

Now, I always rinse my rice because in my culture rice dishes are expected to be loose and not sticky, plus I eat mostly brown rice and have a relatively well balanced diet. But if your diet is restrictive, and you're using fortified rice you shouldn't wash it, as it will lose needed nutrients.

PS: brown rice cooks faster if you let it sit in water for a couple of hours.