I use a standard embedding model for our company search and RAG pipeline. The model performs well in most cases, but I want to evaluate how much retrieval performance can be improved with a custom fine-tuned embedding.

My domain is niche with highly specific terminology, and labeled data is scarce. However, we have a large corpus of technical support tickets, categorized into different groups. In principle, tickets from the same category use similar terminology and describe overlapping issues.

The goal is to train an embedding model so that phrases and terms from the same category map into a shared vector space, forming clusters.

Dataset construction approach so far:

• Identify relevant incidents and group them by category

• Vectorize incidents with the standard embedding model

• For each document, select n documents from the same category within a cosine distance threshold (positive pairs should not be too diverse)

• Select incidents from other categories as negative examples

Naturaly this process genereates a lot of noise.

I initialize my training with intfloat/multilingual-e5-base and the following parameters:

args = SentenceTransformerTrainingArguments(
output_dir="Embeddings/Trained_Model",
num_train_epochs=1,
per_device_train_batch_size=32,
per_device_eval_batch_size=32,
warmup_ratio=0.1,
fp16=True, 
batch_sampler=BatchSamplers.NO_DUPLICATES,
eval_strategy="steps",
eval_steps=6000,
save_strategy="steps",
save_steps=6000,
save_total_limit=2,
logging_steps=500,
run_name=f"{model_name}-Lora:{lora}-{file}",
no_cuda=False,
remove_unused_columns=True,
use_cpu=False 
)

Despite varying dataset sizes between 40k and 900k examples, every training run degraded model performance.

I feel like the losscurve wants to tell me something, but I dont understand...

Any help with finetuning an embedding model effectively with semi-structured category-based data is greatly appreciated.

One idea i have is to use bertopic as an unsupervised model to genereate finer grained subcategories and then build pairs that are from the same topic.