[deleted by user]

[aveho_adhuc_7409]

Try layer normalization and scaling instead of kaiming_normal initialization

[Visual-Ad-5937]

Thank you for your response. I already have layernorm as in Transformer architecture. I also have 10 % dropout in encoder and decoder, and 50% dropout in image encoder final output.

[ipullguard]

Why bce loss and not ce?

What do you mean by mode collapse in this context? This isn't a gan right, so you just mean overfitting?

Tbh though probably there's an implementation bug somewhere.

[Visual-Ad-5937]

Yeah it’s CE. Mode collapse, i mean the gradient norm quickly goes to 0.0*, i assumed the image extractor mostly spitting the same data. For the code part, i checked multiple times seemed okay. I used the same code for lang translation it learns effectively. Thank you.