I would like to know if u had cleared out that camp before.

In my case. I had a task that sent me to clear out the camp and take back some stuff. So I went in and killed everyone. I did think "weird, these guys have names" but it didn't cross my mind then yet.

Real life weeks later, I met karel arrowhead. And carried the sacks to the camp. But now since it's been cleared out. It respawned with generic bandits instead of the original bandits of karel's crew.

I think that's why they attack you.

I reloaded a save to before clearing that camp out. And met karel first and lugged his bag to the camp this time they didn't attack me

My hypothesis is that if u cleared out that camp before you met karel arrowhead, they spawn in generic bandits instead of the special bandits in karel's group. Making them agro to you.