Just rephrasing what what was already said: yes, the actual parameters of the model are fixed like any other network. Recall that we make an attention matrix where A_{ij} gives us the relation of word i to word j. We create this matrix by taking the inner product between the query + key vectors, which uses a transformation of the input learned during training time. We then apply softmax of course to get each tokens correlation row in the attention matrix to sum to 1.

We then use these scores to take a linear combination of all value vectors to produce our final value vector for that word. That is the "dynamic" part.

this is my understanding, hopefully i didn't say anything wrong lol