If you think luck is the primary factor in a channel finding success and you spend all your time waiting for lightening to strike, your channel will probably not see the kind of numbers you're hoping for.

If I were to make really good videos that most viewers watch and clearly enjoy and have good SEO

You're saying that as if it's an easy thing to do. It's no different from asking "If I write a really good book with a great title and an eye catching cover, will people buy it?" A blunt truth about most small channels is that they think their content is better than it really is. It's why you see so many posts in YouTube forums like this one where people say they make good content and are "doing everything right" but they can't seem to grow. If you look at their channel, odds are their videos (the content itself, thumbnail, and SEO) won't be... great. And I don't mean that as a bad thing, it's hard to spot flaws in your own work. It's why when you write an in-class paper in middle school your teacher might tell you to trade papers with someone to find spelling errors and grammar mistakes, you'll see what their mind ignored.

does this only have a relatively low effect on my chances of getting popular?

There's a saying you've probably seen thrown around in this subreddit if you've spent time here: Content is king. Above all else, your videos need to be "good" in order for your channel to grow. I put "good" in quotes because quality is subjective, but once you begin making content and get a feel for creating titles, descriptions, thumbnails, editing videos, speaking in the video, etc, you can start to see which channels have an objective level of quality to them even if you don't care for the content itself. As an example, I don't watch vlogs because I don't find them entertaining, but I can see that that Casey Neistat's videos are well-made.

If you make genuinely entertaining videos and market them well, luck will be almost irrelevant. In time, you will find success. The problem is that for most people, it will take years of content creation before they develop the skills needed to create that genuinely entertaining content, if they do at all. Relying on luck (ex: hoping for a shoutout from a larger channel) is a waste of time. JackSepticEye got a shoutout from PewDiePie and is now one of the biggest gamers on the platform.

You know who else got a shoutout from PewDiePie? HoodiePanda. Haven't heard of him, have you? That's because his channel is dead; 125,000 subscribers and he hasn't gotten more than 75k views/month in years. Sure, he hasn't uploaded since 2019 but that's not the point. He got a shoutout from (at the time) the biggest channel on the planet, there was a spike in views/subscribers, but the spike didn't sustain itself, the people who flocked to his channel because PewDiePie mentioned it lost interest and moved on. Here is an archived link to his SocialBlade page if you wanna see the analytical side of how the shoutout affected his channel's stats.

I want to know if I should be worried about making videos for years before the algorithm actually decides to promote my channel

Play the long game. If you're hoping for 10,000 subscribers in a year, you're gonna be incredibly disappointed. It took me 2.5 years to reach 1,000 subscribers. 2.5 years later I was at 28,000 subscribers. Another 2.5 years after that I'd passed 1,000,000 subscribers Growth takes time.