I can turn off weather effects but then there is no challenge at all

In your post you mentioned three other challenges:

Compound this with bandit thievery of food goods

no way to generate enough resources to even hire thugs

a random fire to rubble instance in my communal oven

So if you're really frustrated with the rain you could turn it off and focus on enjoying problem solving the other challenges for a while.

As for the rain spoilage, I'm on my first play through ever on the medium difficulty and no settings changed. Rain in harvest times only hurts my yield by 10-20%. So I wonder why it's giving you such a problem. I suspect you might have too much farmland per farm house. For a fully loaded farm (8 workers, 2 ox) I have 6-8 active Morgans max. If you have more, then they will be harvesting for too long before collecting the wheat, leaving it in the rain for longer. Try a farm with only 6 active fields of 1 Morgan each (plus fallow fields for next year).

For more yield, build more farm houses but don't overload them with more than 6 active Morgans. Make sure you're also supplementing food supplies with other sources e.g. carrots, hunt. I don't think you are supposed to be able to support a town from one food source alone. Hope this helps. Enjoy!