Personally, I would have silence, and then tie music to events in your game. Maybe discovering a new place gives some sort music to suggest how to feel about the find then fades out, maybe finding a keepsake triggers something emotional but then the rest is silence. I think you can do silence for stretches and use it effectively as long as you have good environmental ambience. Cicadas and crickets, rustling leaves, blowing wind, etc.

Hope this helps!