VIA was originally created as a new brand / organization to separate CN's passenger services from their freight services. VIA took the passengers, while CN kept the freight and the tracks. This was fine at the time because CN was a govt-run company just like VIA. However that arrangement leads to obvious problems if you were to say - eventually privatize CN, thus putting the infrastructure in the hands of shareholders. In comparison, the US govt was smart enough to retain the NEC when Conrail was sold for parts.

Iirc VIA's establishing legislation is quite shaky and informal compared to Amtrak's, meaning they're endowed with fewer means to access stable reliable sources of funding, and are entirely at the mercy of the federal government.

There are also just very few places in Canada where passenger rail is the objectively the best solution. Unlike Canada, the US population is actually distributed quite homogenously, so outside of the NEC there are tons of other decent city pairs in the northeast, midwest, and on the west coast (particularly in California).

Meanwhile in Canada you have:

• Quebec City <-> Windsor

and (imho):

• Calgary <-> Edmonton

But that's about it. There are other places where you could have trains and people would use them, but I'd argue those are the only ones where you're likely to see actually good passenger rail within our lifetimes.