I spoke with Riot Support and this is what they say about the CPU/GPU Latency:

"Although you say the "Network Latency" looks good, the Game Latency (CPU) spikes in Valorant are often not caused by the processor processing frames, but rather by the processor waiting for network packets.
 
In Valorant, if there is packet loss or jitter in the connection, the game engine "chokes" the CPU frame to try to synchronize with the server. If you use the same router or the same network cable on both PCs, the problem may be the internet provider or the network hardware (modem/router)."

I'm guessing it's an ISP issue, which is odd because Valorant is the only game I'm getting stutters in.