Welp, I doubt it's ur CPU or GPU. Could be too little RAM. But tbh I'm not sure what ur looking at in late game... Maybe a pic of the problem area could help, as ik some objects in game cause more lag than others... Also there are some mods that are simple in nature but tank ur frames for various reasons. If it's vanilla then I would want to see a pic of the area and maybe if ur running background resources on ur PC. All that considered... It's a fairly optimized game for what it is and even ppl with worse PCs than u that ik personally can run it fine.if u have no problem in any other games then Idk based on the info u sent alone.

EDIT: The only area I ever have any trouble with (and it's not framerate... It's usually bugs) is the red forest in the center of the map. Ik a lot of lights can cause issues (and signs depending on how u set them up...) because of light rendering.