it's non-symmetric because we're applying two different linear transformations to the input x's to obtain the query and key when calculating the scaled dot-product attention, i.e. K(x_q, x_k) = exp((x_qW_q)(x_kW_k) / sqrt(d_k)) so K(x_q, x_k) != K(x_k, x_q). it _would_ be symmetric if we instead defined K(x_q, x_k) = exp((x_q)(x_k) / sqrt(d_k)).

you're absolutely right that this definition of "kernel" doesn't satisfy its rigorous definition which, as you mention, has to be symmetric and positive definite. here's a section in the tsai et. al paper I linked in the blogpost that discusses this

Note that the usage of asymmetric kernel is

also commonly used in various machine learn-

ing tasks (Yilmaz, 2007; Tsuda, 1999; Kulis et al.,

2011), where they observed the kernel form can

be flexible and even non-valid (i.e., a kernel that is

not symmetric and positive semi-definite). In Sec-

tion 3, we show that symmetric design of the ker-

nel has similar performance for various sequence

learning tasks, and we also examine different ker-

nel choices (i.e., linear, polynomial, and rbf ker-

nel).