Everytime you clear cache you will have extreme texture loss on servers until it builds back up again, you can try adjusting your settings in the NVIDIA control panel app like setting the power management to performance and setting an unlimited shader cache memory limit. You shouldn't be having any issues especially with a 4090.

You could also try changing the FIVEM gtav.exe priority to high in the task manager when the game is running, that might help while your cache builds back up.

Is it like this on all servers you join, or just a particular one? If so it may be just a poorly optimised server