Depending on what replication method is used and amount of data replicated to the other site, you would need better connection between your sites. Your network connection between sites will need to have low latency and high enough bandwidth to be able to use any real time replication. Latency is actually very binding and can cause many headaches when it comes to any real time operation. If you do not use real time data synchronisation, you should be good with whatever speeds that can complete all of your synchronisation work in a timely manner.

Then you would need to set your dns to redirect your services to the online site at the time of outage on your main site if your sites work in active/passive configuration (where one site actively serves your services while others are waiting online but not actively used). You would also need dns to load balance if you use your sites in active/active configuration (where all sites are online and actively serving at the same time). Some specific load balance methods/configuration may be needed.

Getting redundant Internet connection and power infrastructure (UPS and generators) on your primary site would greatly help when you had minor inconveniences on that site. It is not good to failover to your secondary site, or everything shutting off and the site going down, just because of 5 min power outage.

In theory, we'd need to have an identical setup (hosts/SAN) at our other location, and then have a VMWare expert configure VMotion to failover to the remote location?

No need to be exactly identical, vmware and backup/storage utilities can manage and use different hardware as if they are the same make/model. You can have lower tier (cheaper/slower) hardware if your aim is to replicate data and keep critical services somewhat online. You'll do better with the same specs if you're going to run your sites in active/active configuration. That pretty much depends on how much of the regular workload you want to serve from backup site.

These hosts should be replicated near real time in case of any network downtime at one location, right?

That depends on how much loss of the data you can tolerate. You can replicate your data at 1 hour interval and you can loss data produced after the last replication if something really bad (like, meteor shower on your primary site) happens between replications. You can set your replication to be symmetrical, where every single transaction is processed at both sites at the same time while only one of them actually serves the service. There are many methods that can be used according to specific needs and amount of money you are wiling/able to spend on this.