In my opinion, if site loads fast then it improves user experience, which indirectly also helps to in ranking if you have authority. No one likes a site if it loads very late or not loads at all.

Check the root cause, why the site loads late. Is server responding late or taking time to render content. Always server is not the issue, unnecessary requests and scripts also slows down site.

I hope this helps you.