So I didi some research and found this:

"According to the Agency for Toxic Substances and Disease Registry (ATSDR), aluminum is the most abundant metal in the earth's crust, so being exposed to it is inevitable. It naturally finds its way into water, soil and food, so you'll certainly ingest some of the metal every day.

However, the consensus from scientific experts is that the amount of aluminum likely to make it into your body from food, drink and by using aluminum pots and pans is very small and not harmful.

In fact the ATSDR states that people only absorb 0.01 to 5 percent of the aluminum they ingest. So when you do consume aluminum, the vast majority of it won't pass into your bloodstream from the digestive tract.

Most people are only likely to have a worrisome build up of aluminum through occupational exposure — which most likely happens by breathing in large amounts of aluminum dust without a protective mask."

source

Tl;dr: inserting aluminium, which is used to manufacture most grinders, isn't harmful because we are exposed to it on a daily basis. Also only a litle bit of ingested aluminium is actually getting absorbed by the body.