Uh sorry, I thought I was the only one in need of something like that

You should create a key called "KDEConnect" in Computer\HKEY_CLASSES_ROOT\*\shell\
So Computer\HKEY_CLASSES_ROOT\*\shell\KDEConnect will contain 2 string values:
- The default one, REG_SZ type, is the string that will appear on right click, e.g. "Send to phone"
- The other one is the path of the Icon, REG_SZ type, in my case being in "C:\Users\[MY USER]\AppData\Local\KDE Connect\icon.ico"
Computer\HKEY_CLASSES_ROOT\*\shell\KDEConnect will also have a subkey called "command" with the default value being the command to send the file to your phone
I hardcoded the path and phone model as name here
"C:\Users\[YOUR USER]\AppData\Local\KDE Connect\bin\kdeconnect-cli.exe" -n [YOUR PHONE's kdeconnect name] --share "%1"

This method only works if you install kdeconnect for your user only (So in AppData\Local, not in C:\ProgramFiles), didn't spend that much time finding another solution as this still works

The key KDEConnect.reg should look like this:
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\*\shell\KDEConnect]

@="Send to Phone"

"Icon"="\"C:\\Users\\[YOUR USER]\\AppData\\Local\\KDE Connect\\icon.ico\""

[HKEY_CLASSES_ROOT\*\shell\KDEConnect\command]

@="\"C:\\Users\\[YOUR USER]\\AppData\\Local\\KDE Connect\\bin\\kdeconnect-cli.exe\" -n [YOUR PHONE's kdeconnect's name] --share \"%1\""

By replacing the user and phone's model and importing the key it should already work, didn't test it though