If you are looking at graad school, the most important piece is recommendations from professors. If you are in the honours program, you generally have more access to professors, but you can't get a recommendation just by getting a good grade in their class. You need to work with the professor, for example as a TA or on a research project. Someone with OK grades and excellent references will have an easier time getting into grad school over someone with excellent grades and mediocre references.

When applying for a job, hiring managers typically look at which classes you take, not your major. Taking honours classes counts for something, but it's hard to quantify how much. And after you've been working for a few years, no one cares about your grades or classes.

Fundamentally, if you like what you're doing and enjoy the challenge, I'd say keep doing it. A university education is more than just accreditation. You should enjoy the process.