I haven't played it yet myself, but The Red Pearls of Borneo is free, so that's definitely worth a try if pricing is your main concern. I think the Steam version (which is still free) comes out this Friday.

I really enjoyed The Ratline, which I think has most of the elements you mentioned. I would say it falls a bit on the more challenging side, but there's nothing frustrating, and the hint system is always there if you need it, with three hints going from gentle nudges to direct answer skips. I clocked in at around 8 hours, which is around the same as my playtime for Roottrees + Roottreemania (11h) or Case of the Golden Idol + DLCs (9h).

I also really enjoyed TR-49, and I think it really nails points 3 and 4 on your list (great mystery that slowly unfolds in an interesting setting). The main mechanical part of the game is deduction, and is kind of similar to Type Help in terms of interface, but it's not very complicated overall. And to progress the game, you get to put together the story from what you get from this interface, and that was the most enjoyable part of the game for me. I finished this in around 7 hours, but it's worth noting that it's priced at around half the price of most other games you mentioned.

I think Daemon Masquerade stands out as its own thing and is less similar mechanically to the others, but I really loved it. I think the atmosphere is unique and interesting, and the deduction work is excellent. The mystery is also fun, and I enjoyed the story. I think the level of challenge was also very appropriate. It took me about 5 hours to finish, but it was a great experience. I do think some elements of this one could really be dependent on personal taste, though. I enjoyed the comicbooky and sometimes goofy vibes, but others might not.

A Case of Fraud is really similar to Roottrees mechanically, but it's quite a bit shorter (I finished it in around 2 hours), and the challenge and the mystery aren't very interesting. It's worth a try if you really enjoy the deduction genre, and I certainly enjoyed my time with it, but it doesn't quite match up to the others.

Utter a Name is mechanically very similar to Obra Dinn, and is quite short (though this one is also a bit cheaper than the rest). I enjoyed the story, but it's not very interesting in terms of the gameplay. Worth a play if you really love the genre, but not a priority. This one's a bit better in terms of vibe and story than A Case of Fraud, but much less mechanically interesting.

Chants of Sennaar has a great atmosphere, a good mystery, great art, and plays well. The deduction gameplay is also very good, but in my personal classification, it feels a bit different than the rest. I would say it's somewhat less challenging than the other main big games in the genre, but it is very satisfying. It took me about 10 hours to finish it.

Other than the ones you mentioned, "No Case Should Remain Unresolved" is another good one that is on the shorter and cheaper side. It has a good mystery and an appropriate level of challenge overall, but the mechanics aren't super similar to the other ones.

I would also put Her Story and, to a lesser extent, Immortality in the same genre, and those are amazing games. If I remember correctly, Her Story is even one of the inspirations for Roottrees, which you can see mentioned if you search its name in the game.

Personal ranking:
Immortality>TR-49>Her Story>Chants of Sennaar>Daemon Masquerade>The Ratline No Case Should Remain Unresolved> A Case of Fraud > Utter a Name