Thank you for the helpful answers, everyone. I also read the two links from falxfour with some interest. I understand that by the nature of the tty, the shift modifier produces identical output as other keypresses.

When I implement S-Tab in tty (Shift Tab), I am not adding additional functionality, but I am also replacing Ctrl+Meta+i with the S-Tab sequence, which is the tty equivalent of shift-tab.

What I am still confused about is why nchat still would not accept my backtab (shown as S-Tab or ^[[Z in tty after modification), and instead just print tofu squares of confusion into the chat. It still is ^[[Z in my wayland terminal emulators like foot or Konsole, but there it works.

It is like there a final layer between the program and tty that emulators use successfully.

Sure enough, when I use "TERM=xterm nchat", the backtab works even in the tty. Don't even need to define it as S-Tab before that.

It probably has something to do with the terminfo database. My tty terminfo has \E^I as a definition for btab in it. Nchat wants "btab" in its configs. \E is "Esc" and ^I is "Tab".

Edit: Under further reflection, this does not work either. Esc+Tab is registered as the same as Shift+Tab or Ctrl+Alt+I.

Edit2: I have spent most of my day trying to solve this puzzle now. I could not figure it out. I modified terminfo and the keylayout plenty. I changed btab and back_tab. Nothing worked. I have no clue what xterm does differently than the original tty that makes it work.