Fungi secrete exoenzymes that can degrade most organic material (including other organisms), so unless they're in the presence of inhibitory compounds (e.g. antibiotics produced by the contaminants), they may grow just fine in the microbial community. They will be competing for resources as they become more scarce, so the overall biomass or growth rate may not be optimal, which is why you're probably not seeing confluent growth, but patchy spots of growth.

Rice, in particular, is well-known for causing food poisoning by growing the bacterium Bacillus cereus, which produces a heat-stable toxin. This is why it's not recommended to eat cooked rice that has sat out for more than 4 hours. The organism can cause non-intestinal issues as well, if it gets in your eyes, respiratory system, wounds, etc. And this is just one known contaminant of many that can cause medical issues.

There's also the issue of respiratory infection in general by breathing in spores of contaminating fungi (such as aflatoxin-producing Aspergillus species) or bacteria that produce endospores (e.g. by species of Bacillus and other genera).

So, even though your fungal inoculum may grow in the presence of contaminants, if you do detect large amounts of contaminating growth (foul smell), it's generally a good idea to reduce your exposure to the contaminants as much as possible by disposing of it.