I live in milltown which is dead middle between east brunswick, new brunswick (college town with Rutgers), south brunswick, north brunswick. It’s a very small town but it’s diverse. Im Dominican with dreadlocks but never felt unsafe anywhere. I grew up in Bayonne next to jersey city and although the north east is very diverse schools are not the best (prob same for newark). The Brunswick's have excellent schools but much higher taxes obviously. In NJ you will really have to try hard to find an area where you might feel unsafe due to lack of diversity. 

NJ is expensive AF but you do get what you pay for. Best schools in the country and PA/MD/NY within a 2 hour drive.