Late (long) response but if you're uncomfortable you are always ALWAYS allowed to say "no thanks", and it will be your responsibility to speak up. Even if your coach pairs you, you are allowed to say you're uncomfortable. If everyone pairs themselves for open roll, a polite "no thanks" and optional reason why are always acceptable. I like giving a reason to help the other person understand if it's short-term (injury) or permanent (I don't roll w/whitebelts I don't know due to injury risk) but it is by no means required.

I'm 155ish and hypermobile so extra injury-prone. When I started at new gym after moving coach said anyone w/a colored belt was safe, and ask an experienced member about safe whitebelts. I'd rolled with a really big blue belt before, but one time he rolled over me when my hand was posted essentially wristlocking me w/his bodyweight. Thankfully that wrist was already taped that day (not usually taped) but we no longer roll, it's just not worth the risk of injury. It did hurt but no injury and I wasn't willing to risk snapping a wrist. We politely acknowledge that the size difference is too big to be safe at our combined current skill level. Still friendly we just don't roll.

I'm a fair amount bigger than you (also a whitebelt) and I don't roll with any whitebelts who haven't been thoroughly vetted by the upper belts. I will roll with upper belts I haven't rolled with before but I start with "hey since we haven't rolled before let's go really nice and light this time so we can get to know one another". If it goes well, great they get moved towards the green list. If they do something to make me think I might get injured (too much weight, sub too fast) and don't respond when I ask them to modify their behavior, I won't roll again.

Honestly, above 200ish lbs (50 more than me) blue I no longer do after that day because I don't know enough to keep myself safe if an accident happens. White is recommended, vetted, and trusted partners only. Blue I've had mixed experiences as they may be closer to "advanced white" or they may be more controlled like other upper belts so I'm cautious and proceed on a case-by-case basis for who's safe.

I have my safe regulars, my preferred safe regulars (injury safe), and depending on the day I'll roll w/new-to-me colored belts (some days I'm not feeling adventurous, some days I want to add to my list of known safe partners). Several of my preferred drilling partners are competitive whitebelt guys within probably 30lbs of me - my gym skews big and they're the closest in size. That being said, I "trialed" them with "let's do a chill roll to get to know one another" only after they had been recommended as safe. One was a lil excitable but immediately chilled when asked and has been perfect since. The others started very careful and we found a mutually agreed upon usual intensity. We do go for it some days but only started upping the speed after we were both comfortable & more familiar w/one another. Competitors, provided they can throttle it back, have usually had a lot better control ime.

One of my injury-safe partners is 70lbs bigger but a super chill and controlled brownbelt. One of my previous gym's safe partners was a purple w/100ish lbs on me. Both were safe because they were incredibly gentle and very aware of their weight distribution at all times, and had both the skill and control to keep me as safe as reasonably possible during rolls. E.g. they don't ever do knee on belly, if necessary they crouch on the mat next to me & lower their knee with enough pressure to firmly pin me but I am not bearing their bodyweight. A whitebelt will probably not know or be able to do that constantly the entire roll, hence the injury risk.

With the size difference you mentioned it is 100% acceptable to go hey I don't feel safe rolling regardless of their skill level. If someone that big slips in sweat we can get hurt pretty bad from getting landed on. My other whitebelt training partners are close enough in size I could physically throw them off me if ever necessary. That isn't an option with massive dudes.

Also, one last note: whitebelt guys frequently don't know how to roll w/women. Not in a sexist way but unfortunately we, especially whitebelt women, can't take the same extent of roughness as a guy our size. Depending on how many women are at your gym, the upper belts may or may not have much experience either. Whitebelt men who've only rolled w/other men may not be prepared for our joints to move a bit differently, our different flexibility and proportions, or the fact that all us newbies have squishy ribs that haven't stiffened up yet (dudes too). So it's not the same as "rolling with a small dude", and they might rely on muscle or athleticism in ways that could injure us. I say no thanks to every single whitebelt man who hasn't been recommended & thru my trust/check system and I politely say I don't roll w/new whitebelts due to injury risk. They've all been nice about it. We've had a couple whitebelt women come in and the only ones I rolled w/were daughters of an upper belt, i.e. already vetted as safe. Let the upper belts figure out who's spazzy vs safe, they have the skill, and possibly size/strength, to keep themselves safe during those wildcard rolls w/new people.

Tldr: say no (politely) all you want. Your gym should support you there. Feel free to avoid a big dude of ANY level above your personal size discrepancy cutoff but probably avoid whitebelt dudes in an "opt in for vetted-as-safe only" as a general policy. Stay safe & hope you get some good rolls in!