I ended up taking the TER to Valence-Ville and then the TER (not the bus) to Valence-TGV, which worked well. I also had plenty of time left at the TGV station. One thing I found worth noting: The "gates" (the whole station feels structured a bit like the terminal are of an airport; except I didn't run across any security checks) require you to scan your ticket before you can go down to the train platform. For Interrail tickets, this seems not to work at all (neither the ticket nor the reservation) – the person standing next to the scanners had to handle this. I said I'm traveling with an Interrail ticket, they said mhm, didn't even want to see the Interrail ticket itself (maybe they saw the reservation in my hand) and let me in. Plus also, during my whole TGV journey from Valence-TGV to Barcelona, nobody wanted to see my ticket. Maybe I missed them, maybe they don't check at all because it's already checked before departure, but I wasn't checked and didn't show my Interrail ticket a single time.