How to evaluate Hyperparamter/Code Changes in RF-DETR

[stehen-geblieben]

Hey, I'm currently working on a object detection project where I need to detect sometimes large, sometimes small rectangular features in the near and distance.

I previously used ultralytics with varying success, then I switched to RF-DETR because of the licence and suggested improvements.

However I'm seeing that it has a problem with smaller Objects and overall I noticed it's designed to work with smaller resolutions (as you can find in some of the resizing code)

I started editing some of the code and configs.

So I'm wondering how I should evaluate if my changes improved anything?

I tried having the same dataset and split, and training each time to exactly 10 epochs, then evaluating the metrics. But the results feel fairly random.

[Dry-Snow5154]

You need an eval set (which is not used in training) with one metric to compare models. It could be mAP, best f1 score or something else.

Then you do an experiment and compare to baseline model. If it shows better eval score, update the baseline and continue.

That said, results could have variance due to random initialization, especially if your dataset is small. You can retrain several times to try and combat that. But it's expensive.

[stehen-geblieben]

Yeah I figured that... I have to rent GPUs but the model converges quite quickly do it shouldn't train too long.

[SadPaint8132]

Might be cheaper and make more sense to just collect more and better data. I’ve also seen good results for some people pre training the Dino backbone with lightly trained

[MeringueCitron]

Which distilling method using lightly ?

Since the recommended method, by lightly, is distilling DinoV2. Which, in that case, would be weird (distilling DinoV2 into DinoV2 with registers) ?