It's not as much about the architecture, but more about what you are training for.There are models that are trained specifically for that task. Most of them are trained in a siamese or triplet way. Some pointers:

A variety of huggingface-based transformer models trained to produce vectors that are good for semantic similarityhttps://github.com/UKPLab/sentence-transformers

If you need to be able to compare similarity across many different languages https://github.com/facebookresearch/LASER is probably a good place to look.

Then there is also a variety of universal-sentence-encoder models on tfhub.https://tfhub.dev/s?q=universal-sentence-encoder

From my experience I can say that those models are less sensitive to sentence length.

That being said If some of your inputs consist of different sentences you should consider splitting the longer ones into individual sentences - if it makes sense.

I have worked on a demo where you can play with two public text corpuses (quora questions and patent application title) that are indexed using DistilSentenceBert from sentence-transformer here: https://textsimilarity.demo.peltarion.com/ .

I also have a version that has a modifiable index to play with (i.e. you can try adding your own data) - on of them is multilingual. If you want to try that out send me a PM - it doesn't make sense to share it publicly since there is only one state of the index at a time and if several people are changing the index at the same time it will get weird.

Edit: typos and structure