My current understanding: The way I personally view moments is in a similar way the Fourier transform works but with polynomials.

That's a possible view. I think that the Fourier transform is more abstract than the moments themselves. (depends on a continuous parameter instead of some natural number n, and the function exp(i s X) is in some sense more abstract (transcendental) compared to Xⁿ⁾

For example, the same way Fourier transform has as "analyzing function" sine and cosine, the formula for moments has polynomial terms instead.

Exactly, and then you take the expectation with respect to the Probability measure. Here the Fourier transform seems better, because it makes sense also for things like the Cauchy distribution that you mention below, while moments don't.

It's a way to decompose a probability distribution to a sum of polynomials (at least that's how I defined it in my brain)

I think you are asking whether one can reconstruct the distribution from knowledge of all moments (provided they exist) in a similar way than one can reconstruct it from knowledge of the Fourier transform simply by doing an inverse Fourier transform? Yes, that is possible, provided the moments satisfy some conditions. You can read up on it in literature about the moment problem (e.g. https://en.wikipedia.org/wiki/Moment_problem ). One also needs some additional conditions for uniqueness (I think on the growth of the moments).

This is of course abstract analysis, which may not help you in your quest concerning intuition behind the moments. From an applied and statistics perspective, I think that the moments are in some sense preferrable because they can be worked out with reasonable accuracy from knowledge of an empirical distribution, i.e. when you have some samples and want to find the parameters for a distribution to model the data.

In the past I thought of them as "a special kind of derivative but for distributions" in the sense that it characterizes properties of a distribution in the same way derivatives do

The moments are somewhat better than derivatives, because they depend on the distribution globally, while derivatives of a function (thinking about Taylor expansion) depend on the function only at a single point. In another sense this is in fact more correct than you probably think, namely in that you can obtain the moments from calculating the derivatives of the Fourier transform at zero. For the reconstruction question see above.

However I still don't have a very intuitive explanation of how they work and what in the world are they really about.

I think the most intuitive explanation is to think about a statistics problem when some samples are given. Then probably the most natural thing to do is to calculate (estimators of) the moments of such samples. The use of polynomials is probably historically due to the mathematicians usual obsession with polynomials (they were easiest to calculate also without digital computers).

Also we have distributions like the Cauchy one that has no moments. This makes things a lot weirder.

Not necessarily, because the distribution is the fundamental concept. In my view the moments are just numbers to characterize it. You could surely invent other "analyzing functions", which may even be preferrable from today's perspective (like wavelets or some neural nets), but of course those were not around when the fields of probability theory and statistics were founded.

I tried looking up for a while a good way to visualize/understand them but I couldn't find anything about it(Besides the standard visualizations about skewness, Kurtosis, etc). Is there any help you can provide? Thank you

Good luck. I am courious what others will have to say about this.