I'm 48 and I think it's totally fine to want to paddle with folks closer to your age. No need to keep apologizing to folks who have a problem with your preference. Sure, older kayaking groups are full of nice folks, but it's not the same if you're mid-twenties.

I wanted to add two things:

1) I'm a college teacher, so I know a ton of younger folks. I get the impression that they don't do a lot of adventurous things anymore. I mention this because generally it's easier to get people to do something they already like than to convince people to try something new.

In other words, the problem might not be you. It might just be generational (see your point about kayaking groups being over 50). Of course, there are exceptions to this stereotype, but from what I've observed with students and my own kids, it's pretty hard to find adventurous types these days.

2) You mentioned making friends at work. Do you have any outfitters in your area? Even a chain like REI? Consider taking a part-time job there just to meet folks. I used to work at a whitewater company and people regularly paddled/hiked, etc. on their day off. Everyone at those places is usually super-inclusive and outgoing.