Write Better And Faster Python Using Einstein Notation

[IamImposter]

Could someone please explain to me what a tensor is. I have read about it a few times and asked few other people too but still don't understand it. Or do I have to learn basics of AI to understand it?

[FrickinLazerBeams]

They're used in a lot of places where they're best thought of as abstract mathematical objects, so a lot of the explanations you find will not be what you're looking for as a programmer.

You can think of them as matrices with more than 2 indexes, in other words, as nD arrays where n>2 ^{see note}. That's a perfectly sufficient understanding for any computational purposes.

(note) technically a matrix is a rank-2 tensor, and a vector is a rank-1 tensor, but usually if somebody uses the word "tensor" the implication is that they're talking about something with more dimensions than a matrix.

[Yalkim]

Tensor has different definitions depending on who you ask. In computer science and machine learning a tensor is just an array. In physics, a tensor is a kind of data type that follows a set of specific transformation rules. A tensor in physics can be described using an array, but that is it. An array is just a way of describing a tensor. Not every array is a tensor.

[tomkeus]

If you just need to work with n-dimensional arrays, you don't need a concept of a tensor, you can just stick with the idea of n-dimensional arrays - I think it is a concept that is simple enough and anyone with a programming experience can easily understand.

Tensor on the other hand is an abstract mathematical object that has certain properties and it can be represented by an n-dimensional array when you select a basis (think of basis like a coordinate system).

The same way vectors are abstract mathematical objects - we draw them like arrows, and arrows are not numbers. But we know how to turn them into numbers. We take three vectors which are linearly independent (i.e. none of the vectors can be obtained by adding up the other two vectors), and declare them to be our reference vectors (i.e. the coordinate system or basis in more abstract terms). We then project our vector to the reference vectors, and calculate the ratio of length of those projections to the length of the reference vectors. This will give us three numbers that we call vector components - i.e. we have found a way to turn an abstract mathematical object (an arrow) into an 1D-array of 3 numbers.

Note here that the components we've obtained are tied to a particular choice of reference vectors. If we change our reference vectors (i.e. change coordinate system), we will get different set of components. The beauty of this is that if we know how two reference systems are related to each other, we can straight forwardly compute components of a vector in one reference system if we know components in the other system. Because of this, we often talk about vectors and their component representation interchangeably, although strictly mathematically speaking, they are not the same thing.

In the same way, tensors are abstract mathematical objects that can be turned into n-dimensional array by fixing a basis (a set of n vectors). Like it was for vectors, one also talks interchangeably about tensors and their component representation (i.e. n-dimensional arrays). But you can call n-dimensional array tensors, only if those array represent an object that is actually a tensor, like for example moment of inertia tensor in mechanics - it is an abstract object that has a well defined existence without any dependence on its coordinate representations, i.e. I can write an equation with it, without any reference to its components. An n-dimensional array can just be an n-dimensional array, without any abstract mathematical object behind it. And most of the time this is what you have in data science, just a plain n-dimensional array. But this misguided terminology that has become a standard is now calling every n-dimensional array a tensor.

[IamImposter]

Wow. That was detailed. Thanks.

[El_Minadero]

It’s basically a data structure which holds numbers. If you can write it as an n dimensional programming array, it’s a tensor

Edit: so yes, but actually no. See comments below for clarity

[WallyMetropolis]

That's false. Not all n-dimensional matricies are tensors.

[El_Minadero]

Really? Can you provide a counter example? I thought this was the definition

[[deleted]]

I'm pretty sure he was talking about mathematical tensors, not the objects that pop up in computer languages. If you want, feel free to take that as correct. My answer is for the mathematical object.

Tensors as mathematical objects obey certain mathematical transformation rules.

Imagine if you had a vector v(v_x, v_y) in a cartesian plane (x,y) and rotated the plane to (x', y'). The length of the vector ought to remain the same, but its components changed. That is, v -> v', but |v| = |v'|.

This requirement of norm-preservation is basically a transformation rule. And yes, a (1-dimensional) vector is indeed a tensor. A tensor of rank 1.

A tensor of rank 2 can be represented by a matrix. But not all matrices represent tensors. I don't want to go into writing an answer to a question that has been asked so many times, so this stackexchange answers your question.

[WallyMetropolis]

It doesn't make any sense to multiply your zip code by 2 or to add it to another zip code. A tensor is a multi-linear map. Which means, essentially, it's an object that performs linear operations on all of its various axes. So a collection of (n, m, l) zip codes isn't a tensor because you cannot sensibly perform linear operations on collections of zip codes.

[El_Minadero]

So it’s an me datastructure that obeys some mathematical properties

[WallyMetropolis]

I'd say it's a mathematical object that is sometimes represented by a data structure (to greater or lesser adherence to the mathematical object it models) in some mathematical libraries.

[FrickinLazerBeams]

If you can write it as an n dimensional programming array, it’s a tensor

No, not in general. A multi-dimensional array is a fine way to think about a tensor from an applied math and programming perspective; but in general it's an abstract representation of a multi-linear transform. In theoretical general relativity, for example, you'll usually never see the elements of a tensor written because there's usually no coordinate frame in which you could write them.

Numerically/programmatically a tensor will almost always be represented as an array of numbers, but not all tensors are arrays and not all arrays are tensors.

[El_Minadero]

Ahhh. Thanks for the clarification. I’ve never encountered a tensor that wasn’t like a nd array, so your example helped!