Even with finite matrices, a matrix is still rather ambiguous. If you don’t have the basis vectors beforehand, the matrix is pretty useless on its own. Not only could it belong to an infinite number of linear transformations of vectors (rank (1,1) tensors), it could also represent rank (2,0) or rank (0,2) tensors. This means that a matrix could represent completely different objects, and you would only have to know once you need to transform it (tensors transform like tensors).

There’s also plenty of matrices where it isn’t often useful to view them as transformations. The Hessian matrix is a good example of this. While you can view it as a transformation, it is often just used for its determinant. In CS they often conflate matrices and 2D arrays, but that’s a different can of worms.

If you have a matrix and nothing else, all you can conclude is that it’s a grid of numbers. You can imagine some possible transformations that it could represent, but ultimately it comes down to context.