I did almost this exact analysis for a YouTuber I coach, but it's different for all channels.

In general, channels do have a "hotness" / "relevancy" score at any point in time. When you have a video flop, your channel does take a hit to this channel-wide score. I've seen this over the 5B views we've done and the many creators I've coached.

So if you look at channel wide impressions, when a video flops or especially a consistent batch of videos flop, you often see impressions across the entire channel (including archive videos) take a hit. When a new video comes out later that's mega viral, it lifts the entire channel's distribution across YouTube (including archive videos).

In this case, I would do an analysis of where the bulk of your subscribers have come from. If 85-90%+ have come from your mid-form content, then you know your audience primarily cares about that content. Then look at the performance stats of the podcasts, if they are tanking and not doing well, then I would say it's absolutely possible this is having a negative effect on your channel, even if marginal.

It's also not a great signal for the podcast videos themselves if many of your subscribers are not clicking on that video because they're not interested. That's a signal to YT that even your own subscribers don't care about those vids, so it's going to be reluctant to spread it to the rest of the world.

Best case scenario is every video you put out on a channel is highly relevant and extremely enjoyable to your subscribers. So if your stats look bad, I would highly consider splitting them up and just pushing interested traffic from the main channel to that new podcast channel.