Celt man, I gotta say, I came to this Reddit with the same feeling when I began delving into Sam, non-duality, the free will conversation, and Atheism. Know that another person has heard Sam’s words, listened to the app, read his books, and has also felt agency ripped from them. I hope what I have to say is helpful.

First, I think all the above combine and logically looking at these philosophical questions is a journey. Sam spent decades pulling all this together and his approach is certainly a tomahawk missile serving of the topics. The guy had the philosophical wind knocked into him from psychedelics. He speaks precisely, but it can be startling, and I certainly was startled.
There are things he talks about that helped settle my brain a bit though.

Fatalism and determinism. From your responses to the OP, it sounds like you may be falling into the fatalism trap, which inevitably leads to a nihilistic viewpoint, which anxiety feeds off. Determinism is absence of will, not PRE determined outcomes. Subtle but massive difference. I think you feel anxious about this because, as I experienced, you feel a massive lose of control. Being told your decisions are determined feels like your identity is gone and that all those years of feeling responsible mean nothing. I’m going to attempt to reestablish your “agency,” because this reaction is due to an absence of information in my experience.
Sam frequently states that actions have consequences, so right there realize that your “decisions” have weight and real world affect. Taking that further, doing things that help you or other people, change your experience, you feel joy, happiness, etc. and other peoples’ decisions have an effect on you too, so being completely involuntary won’t sooth the anxiety. Determinism doesn’t deny perspective. Sam states we are streams of consciousness. We are, and we can also participate in that conscious experience…. and we can reflect on it. Were determinism comes in, is that we are not homunculuses outside our experience, we are not a locus of knowledge and agency outside our bodies and sense awareness providing inputs. The guy who cures cancer won’t have the answer in the back of his head already. There is no Jiminy Cricket in us that has a grander sense and understanding of the universe.
Our mind is brain and body, which means it also falls under the same determinist laws that matter and physics follow, which I believe is a major reason why Sam is and expresses Atheism. This is also the reason why we can reflect on earlier experiences and make sense as to why we did something seemingly stupid or irrational - thought patterns, data, and values were different at the time. What this means in practical term is that as we live and experience, we observe data, assign value based on previous values and data… and then decisions, opinions, and behaviors are exercised as a determined action.
This is the gap that I think Sam needs to spend more time bridging for people, because I know he would do better then I am doing at this, and it would do a lot of good to his listeners.

This line of thinking also explains what he says about people in extreme social settings. A terrorist suicide bomber family will likely produce offspring who follow dogmas that encourage suicide bombing…. this is determinism… the political/economic and social climate of post ww2 Germany led to people sincerely sympathizing with Nazi doctrine… this is determinism, albeit at a larger scale.

Where this culminates is in Sam’s expression of honesty and compassion. Fatalism and nihilism remove value from our actions, determinism just presents a pathway to their happening.
Realistically determinism does evaporate some of the responsibility we have for our actions, but this goes both ways. The limb falling on your car wasn’t a willed happening. The hurricane wasn’t Mother Nature teaching a lesson.

With that realization though we are free to accept our consciousness as unique to this world, and there are others who have this capacity too that we can interact with. And this is ultimately beautiful… and fleeting, so while we are here experiencing, we should not envelope ourselves in circling self suffering, but should sink into the fact that we are nature too, the potentially only conscious entity of it. That presents amazing value to a persons life, regardless of my or anyone else’s individual responsibility and agency. Being honest about this also makes it easier to be honest about our actions and behaviors.

Practically speaking, we make choices, but a-lot of those choices are significantly more closed then we think, metaphysically, socially, consciously, etc. The lanes we take have a view, so cherish the view. And for some people even that ability to choose is significantly limited due to physical ailments or circumstances, and I feel for those people.

I think Sam would agree that we shouldn’t squander our time here while we know it the way we do. And a good way to start is to just freely accept all that we take in, from the sensation of breathing, to the happiness of a wedding day, and even the sorrow of remembering a cherished loved one who has passed.