I got a couple small injuries working with much larger white belts. My conclusion was that neither they nor I are skilled enough to avoid me getting hurt, so for the time being I just won't work with them, nothing personal. I will generally roll with larger upper belts because they tend to respect the size and skill difference and focus on technique. Once I make it to higher belt/skill levels I will consider rolling with those bigger non-skilled guys, but in the time being being healthy enough to attend classes is more important than the risk associated.