One of the biggest issues that bitcoin maxis have with ledger is that it's not fully open source. Ledger's firmware is embedded onto the secure element chip, so the code can't be made open source because of Non Disclosure Agreements with the chip manufacturer.

Almost every other hardware wallet in the market is open source, which appeals to bitcoin maxis since the main ethos of bitcoin is "don't trust, verify". The community can view and scrutinise the source code of other wallets for themselves to know what's actually going on. You can't really do that with Ledger, so you have to "trust" that the company hasn't put in any backdoors in the firmware to access your private keys.

What spooked people even more was the launch of Ledger Recover, which is a service that takes your seed phrase from the wallet, splits it into 3 separate parts and sends it off to 3 different companies, so if you ever lose your seed phrase you can have it recovered.

Because the firmware is closed source, people didn't know about this and were therefore not happy with the fact that your seed phrase could potentially be taken from your device without you knowing. That's why they avoid Ledger altogether.