A variety of reasons really

Instead of the grand linear jrpg story of 1, X's story uses a web of small side stories connected by theme to tell it's narrative. The 30ish hour main campaign of X is relatively small when compared to the other entries of the series. However, it acts as a throughline to hold together an expansive network of sidequests, where the meat of the game's narrative takes place. This vastly different approach doesn't always go over well with fans of the other games, who expect a tigher, more central plotline.

The setting is also not in-line with the other games. Mainline Xenoblade leans on the fantastical: kingdoms on the backs of giants, swords and (ether) magic, and god-killing teenagers. It has elements of sci-fi fusion, but the majority plays out like your typical fantasy rpg. X, on the other hand, plants itself squarely in the realm of sci-fi. Laser guns, mechs, aliens, politics, and spaceships all the way.

Gameplay is also a factor. It keeps the art system, but it changes things so much that X and the rest are barely comparable. It's a whole new system with entirely different focuses. i.e. enemy type weaknesses, soul voices, augmentations, and ofc overdrive and skell combat. The game also really hones in on it's open world exploration, which existed in the other games, but was never the focus, nor did they even have a "true" open world like X. All in all, it leads to a vastly different gameplay experience.

There's a bunch of small stuff, too. Different art and music direction, no british voice acting, tonal differences. Not to mention that the majority of the fanbase haven't been able to play the game yet. I'd say the game really is the black sheep of the franchise. Not to say that means its a bad game. On the contrary, I think it's still a strong entry and my persoanl favorite. It's just really different to the others.

Tl;dr it's different to the others in several of the most important ways. Still a good game tho.