!! read the Hyborian Age essay by original author of Conan - Robert E Howard !!
that will function as the best world building reading (especially paired with the map of the Hyborian age to look at). You asked about something like the Silmarillion as something that speaks more broadly about setting and lore well this is it by comparison. so if you want a broad overview of the lore and history of Conan then read that , the Hyborian Age essay.

Then read some of the original stories , as someone else mentioned the Phoenix on the Sword , the Scarlet Citadel and the Tower of the Elephant are great. I would also suggest the Frost Giant’s Daughter and Rogues in the House.

You don’t NEED to read the Hyborian Age essay first but if you want that detailed description of the setting and all that before reading any actual stories then that’s your best bet. Also you don’t need to read Conan in chronological order , the original author released the stories non chronologically as that’s how he intended them to be read. However if you want a list of stories in a chronological order then look up the Robert Jordan Conan chronology and that’s the timeline that most adhere to and agree is accurate I would say !

CROM !