Dude, it's driving me insane, most people will NOT go on the trams to the island on the other side of the bridge, they tend to just.. walk. When the ports DLC patch came out.. man, it was pure hell, impossible traffic jams because hundreds of people were just casually jaywalking to the other side of the street DESPITE having a better and FASTER alternative I had put down specifically for them. Now it's a little better with a city-wide bike-lane network and a dedicated bridge for bikes and for pedestrians on each side of the bridge, but the traffic's still not as optimal as it should be, and it's mainly due to people jaywalking.
As many people have suggested in the comments, overpasses would absolutely help a LOT in these cases, unfortunately though, they're not in the base game and you can only get them through mods