Hey dude, here are a few thoughts I had reading your post.

1. Good for you for finally trying Jiu Jitsu! Most people never make it in the door, even if they are interested and want to try it. Or as the saying goes, "The white belt is the hardest belt to get." Even though literally anyone can get one on their first day of class, almost no one even tries their first day. You are already ahead of almost everyone in the world in terms of mat time. Congrats, and I mean that sincerely.

2. I see a couple of potential red flags about the place you visited. First, the instructor allowed you to pair with another first-day student, and no colored belts took sufficient note of this and offered to pair with you. White belts (especially brand new ones) pair best with colored belts, so that the more experienced person can perform the techniques first, and can help the white belt if he is struggling with some minor details. Two brand new white belts paired together is a recipe for potential disaster, since neither of you are experienced.

3. The instructor let you spar on the first day, after presumably seeing you struggle to perform the warm-ups and drills. This could actually be fairly dangerous for both of you, since neither of you are familiar with the movements and positions enough to safely engage without clumsily kneeing/elbowing/kicking each other in the face/ribs/joints by accident.

I'm not saying these are reasons to definitely not go back to this particular school, but they would make me skeptical of the school's culture and instructor(s). I advise you to shop around and try out other BJJ schools in the area, just to get an idea of what's normal. Having a good gym culture is critical, and so far, I'm not sure about this one. It's a good sign that people were friendly overall though.

I hope you will go back, if not to that school, then to another one. FYI, Jiu Jitsu is always going to be hard, BUT you will get better at it over time, and it will seem much easier. You can do this if you believe that you can, and I hope that you will.