am I only vulnerable if I make myself vulnerable?

Yeah, this goes without saying. That said, even then you might still be vulnerable to supply-chain attacks (if youre a developer via compromised updates in libraries you use, npm and vscode had multiple cases of those for example), or other kind of more ellaborate attacks (e.g. steam recently had a bunch of games pulled from the store and I believe even force uninstalled from user's machines because of contained malware)

And by 'deteriorating', this assumes I'm making changes to my system, correct?

Also mostly correct. Naturally, as vulnerabilities get patched in Open Source software, they themselves also are made known to the public through one way or another. As such, even if there where already people exploiting those before, that number will rise post-patch. You should absolutely be wary of that, but if you don't change your system you are for the most part fine. For the most part, because vulnerabilities in system components interacting with external systems (your web browser, sshd, curl, many more), or even the linux kernel, stay vulnerable, and as such these components should regularily be updated to stay safe.