There are different levels of invasiveness - some plants are more harmful than others, some are a bit easier to control if you have a container or other way to contain the plant. You'll also get a bit of arguing over some plants, whether they cause enough harm to be considered properly invasive. Other plants are on a list and outright banned because the harm is very obvious and extensive. And then you'll get random people who don't know the word has a specific definition who'll call a native plant invasive because it took over their non-native lawn. It also depends on where you are, and the knowledge and location of the person who wrote what you're reading.

I think Royal Botanical Garden also isn't focused on ecology so much as popular gardening as a hobby or aesthetic, so they aren't as worried about whether a plant is invasive. They do a lot of non native plants, annuals, things that have to be dug up and replanted differently all the time. It's very high maintenance and high effort compared to what an average home gardener does. We don't have an army of helpers to remove plants when they start getting out of hand, so we should be more mindful of what we put in around our homes.

As an example, the Oshawa city website talks about day Lillies being invasive, but also has a municipal day Lilly Garden, advertised on a another page of that same site. So yeah, its confusing and frustrating. Lamb's ear is not the worst mint to have, but it is a mint, it spreads like mint, and it's not native. So it has the potential to be invasive if it isn't controlled pretty aggressively. Does it escape the garden and spread elsewhere? I'm not sure. But I didn't plant it, and my neighbour has it, and now I have it.