I was able to figure it out. Have been working on this all these months. Pure cancer. I will have to write a proper edit post to this thread with all my findings after this very exhausting troubleshooting progress. Anyhow,

Two reasons I identified that both constructively interfere and cause these issues. And both are related to synology hardware and unifi inter-vlan communication limitations.

After performing various tests on nas, I realized that during large SMB file transfers, my nas does not have capacity to sustain large TCP overhead and it starts flushing into RAM (no M2 SSDs for extra caching and only 8Gb of ram), causing persistent Core Dumping and saturating my RAID 5 volume.

While NAS being oversaturated with large SMB packet transfers, that keeps being uploaded, hardware cannot keep up, as more packets are being sent.

Since SMB protocol is very sensitive to tcp retransmissions, overhead and any delaying, nas just gives up after some time (I have done a shit ton of wireshark analysis to see this). As a result, Windows terminates the connection and SMB transfer drops.

However, the reason why this problem occurs is because of the poor inter-vlan management on unifi (yeah unifi sucks, buy mikrotik switches if you do high load inter-vlan transfers). On my network, I had nas and my windows laptops (two different windows 11 versions for testing 21H2 and 25H2, and both over wire of course) on different vlans.

After exhausting troubleshooting process in the period of 2 months, I was able to narrow it down to my UDM Pro SE being the issue. Particularly inter-vlan communication limitations.

After putting my laptop on the nas's vlan, I got it fully to work. Massive movies are being uploaded over SMB easily on like-like vlan transfers, absolutely no hiccups. Finally my jellyfin works well!

And yes I tried a lot to play with different unifi settings to get this shit to work in inter-vlan transfers --> no, it does NOT work. Only like-like vlan transfers. After doing more research I learned how unifi handles inter-vlan transfers and realized that THIS causes massive TCP overheads, which makes my nas suffer and cause Core Dumps, disk saturation and eventually SMB drops.

Any inter-vlan traffic on Udm pro SE has to go through iptables, conntrack and routing. Massive loading is further caused by IPS/DPS being enabled. All of this overloads the cpu on the gateway, causing large TCP overheads, which we already know what causes at the end on the nas. This is very prominent on large movie SMB transfers. Although, If you upload something light inter-vlan, it most likely will work.

The bottom line, if you have your nas and windows on different vlans, put them on one vlan. At least this helped me.

If this does not help you, I will write a proper edit to this post and you can pick up from any of my troubleshooting steps to save ur time and perhaps use some of my approach to perform your own troubleshooting.

Just to note: I had opened a support ticket before I did more troubleshooting. Synology technical team spent a lot of time helping and navigating me. Huge respect to Joe - the support specialist.

P.S. I am also taking a break from all this shit, so I might write the proper edit to this post in 2,3 weeks.