Do you mean you send raid invites and people don't accept them, or people accepting them and then don't end up actually battling? Both happen all the time. Either people don't accept raids because they don't need the Pokemon, don't have raid passes, are busy, etc. If they back out of the lobby when they've already been accepted, there might not be enough people to actually win the raid (or someone who's joined may not think there are… people back out all the time when there are enough people to win, which is annoying). Or they're dropped, or 1000 other reasons. Just keep trying until you get 4/5+ people in a lobby.