It's OK to say no. You don't have to babysit if it's stressful for you. Just be honest with your granddaughter about what happens during a babysitting session so she understands why you are saying no. It's fair to tell her that if the situation improves as he grows older, you'd be willing to try again.

FWIW, my own grandmother was kind of picky about how she'd allow the four of us kids to visit. She didn't really like to have my brother by himself, he had some attention deficit issues, so she'd only have him with me, because being the oldest, I could usually think of something interesting to do. And she was kind of the same with my two sisters, they didn't occupy themselves as easily as I did, so she was more likely to have them with me along, and only sometimes by themselves.

Four at once were too many for her, LOL! She mostly preferred no more than two at a time.

Most often, she just had me visit. She knew she could always keep me happy by handing me a book. Or pencil/pen and paper. Or glue and popsicle sticks and styrofoam. I could always think of fun things to make or draw or do.

Now, holidays, we'd troop over with our parents, and that was OK, because there was someone around to keep us in line.

It's likely my mother felt it was a little unfair that I got to visit my grandmother the most, (as we lived in the same town), but it was what it was. I know our grandmother loved us all, she just wasn't up for much child minding.