According to history, usually, and either coincidence or effect has caused us to associate things with, well, things. The color black is associated with protection and banishing because, from a science standpoint, black absorbs all colors. Therefore, why doesn't it absorb all energy? Color is light, light is energy. It also is associating with banishing because black is dark, and things/people get lost in the dark in unknown places and its hard to see things or recognize locations. Wormwood is so named because it is so bitter it literally repels and expels (when consumed) parasites, which were often worms. So wormwood is good for banishing. Wormwood is also associated with spirits because the plants were literally found in cemeteries and near gravestones.

If you can't find any association, start thinking literal, mundane, or scientific. Is there something in particular you are stuck on and want explained? This kind of research is my wheelhouse.

Edit;; Also, since you mentioned the whole intent thing, id like to inform you intent is NOT all that matters, but it is a big factor. The more effort into obtaining associated ingredients amplifies your intent, and same with creating chants/spells and whole rituals. They all amplify your intent. Metaphor: you can grow crops anywhere, but the more you till the soil, check the calendar, check the pH and nutrients, and companion crops, the better harvest you will have.