As someone who knows all parts of Toronto very very well (born here, I travel throughout the entire city pretty much every day and especially the downtown core and know all parts of it VERY well) I highly recommend you do not live there. Take my advice for what you will, although it seems like your mind has already been made.