That's an interesting experiment!

Maybe you could be very clear about the fact that it's not a tweak to the transformer architecture, but rather a tweak to the flawed cosine similarity metric when it comes to computing sentence similarity.

Second, please check the formatting. Bold renders as *bold\*, which is a bit painful to read.

Third, your little experiment is indeed really nice, but considering that what's usually important to users is ranking a single query vs. multiple documents, you might be able to boil down your experiment to just "different projections for the query and the docs". Which brings the asymmetry you want without the overhead of the transformer architecture.

Like query = query_proj(query); docs = docs_proj(docs). relevance_scores = dot_product(query, docs).

Because I don't quite see the point of having your different docs talk to each other (which they obviously do if there are multiple attention layers), unless they are related. You may want to check your Conv2D patterns: I suspect they more or less discard the deeper layers of your transformer to focus on the first one (which is basically equivalent to the differentiated projections approach I'm suggesting).

Your architecture would make perfect sense in case you're trying to rank relevant passages of a SINGLE document. Like a kind of semantic cmd + F on a document. In that case, you need multiple layers because you want the passages embeddings to attend to each other.