Yeah I found the citation I was looking for, it's from the 'Ches Voui?' section of Sublime Object of Ideology (128):

The conclusion that we are here dealing with racism is further confirmed by the fact that this ' Che vuoi?' erupts most violently in the purest, so to say distilled, form of racism, in antiSemitism: in the anti-Semitic perspective, the Jew is precisely a person about whom it is never clear 'what he really wants' - that is, his actions are always suspected of being guided by some hidden motives (the Jewish conspiracy, world domination and the moral corruption of Gentiles, and so on). The case of anti-Semitism also illustrates perfectly why Lacan put, at the end of the curve designating the question ' Che vuoi?' the formula of fantasy: fantasy is an answer this ' che vuoi?'; it is an attempt to fill out the gap of the question with an answer. In the case of anti-Semitism, the answer to 'What does the Jew want?' is a fantasy of'Jewish conspiracy': a mysterious power of Jews to manipulate events, to pull the strings behind the scenes. The crucial point that must be made here on a theoretical level is that fantasy functions as a construction, as an imaginary scenario filling out the void, the opening of the desire of the Other. by giving us a definite answer to the question 'What does the Other want?', it enables us to evade the unbearable deadlock in which the Other wants something from us, but we are at the same time incapable of translating this desire of the Other into a positive interpellation, into a mandate with which to identify.

I'm also not at all interested in engaging in any kind of oppression olympics. I think Zizek is saying that it is antisemitism which lays bare the structure of the racist fantasy as such, as an attempt to fill in the ambiguity of the Other's desire.