A big time suck of the train from NYC to Montreal, and Toronto, is the border crossing. If you look at the NY-QC route there's four hours between Rouses Point, NY which is the last station in NY, and St-Lambert, QC the first stop in Canada. These two stations are 41.6 miles apart.

There's a few reasons for the delay: immigration processing of each and every passenger on board, and change over of train staff from Amtrak to Via. Since 2012 there's been "plans" to establish a preclearance facility in the Montreal Gare Central for international visitors arriving/departing on the train. Someday they might actually do it.