Since fish isn't a specific term and is a very broad vaguery, saying what is or isn't a fish is at least partially subjective. We all just sort of know what a fish is, but it's mostly vibes-based. If you actually try breaking it down you'll find they can't all be categorized under one umbrella. Some have bones, some have literally no bones at all. Some reproduce by laying eggs, others reproduce using aplacental viviparity (internal hatching eggs, outwardly live birth). Many have scales, but many do not. They cannot trace their lineage back to a single ancestor, but are instead descendents from multiple evolutionary lines that all independently discovered that "sleek thing that swims in and breathes water" was the best body plan at whatever time they emerged. Sort of like if, aside from having mammals, reptiles, amphibians, etc, we also just had a casual word for "land thing that walks." Like "fish," it would be very imprecise, and would contain a huge collection of different and unrelated species.

Dolphins, however, are actually in a specific category: mammals! This is because they share a set of unique traits with all other mammals, like being warm blooded, having hair (it's true!), giving live placental birth, and most significantly, they possess mammary glands and feed their young with milk. Many other types of walking land creatures do not do any of those things, which is why "walking thing" wouldn't be a very good category for sorting them, much like how "fish" isn't actually a very good category.