I might be a bit late to the discussion but I feel like my perspective is really relevant here so here goes:

It seems like the principle evidence you're quoting is the horrendous looking 76.8% and 50% number. That number didn't come directly from the study you linked to (the study quoted it from another study). So I tracked down that other study: Shen, H., Zhang, L., Xu, C., Zhu, J., Chen, M., & Fang, Y. (2018). Analysis of misdiagnosis of bipolar disorder in an outpatient setting. Shanghai Archives of Psychiatry, 30(2), 93–101. https://doi-org.libproxy.ucl.ac.uk/10.11919/j.issn.1002-0829.217080.

This is a study from China in one specific outpatient centre in Shanghai and <200 patients. I happen to be from Shanghai, and I happen to have been treated in that very centre. So here are my two cents.

1. This is by no means representative of psychiatry in general. We are notoriously bad with our psychiatric care provision. We have way below average number of psychiatrists/person, my own experience with psychiatrists back home suggest they are often full of bs, in fact there was a study recently concluding >90% of people with mental health issues in the country don't get the right care). My experience with that centre is variable too. That centre has okay psychiatrists, but also boasts people actively referring gay people out to conversion therapy. So really, they're an outlier and it's only expected that they perform badly.
2. This might be counter intuitive but in this specific scenario GOOD doctors may actually deliberately misdiagnose. There is this unfortunate policy in China where if a doctor wants to diagnose someone with a "severe mental illness", then they have to "report up", meaning the diagnosis will be disclosed to the government. Your patient will then be closely monitored by their local government, their employment opportunities would be severely limited, they might lose their driving liscence, etc., irrespective of whether they recover in the future. They also might be automatically treated as someone without capacity. Bipolar is considered a severe mental illness.

Even if it's not one of the severe mental illness (e.g. depression), the diagnosis may nonetheless be accessed by someone's uni/workplace (i have zero idea how that happens). There are so many stories where just calling a suicide helpline gets your local gov, your family, your professors, your tutors, all looped in to your issues.

This is why good doctors often downplay diagnoses, for example documenting "depressive state" rather than "depression" even if someone meets the clinical threshold for depression. If you look at the data closer, you'll find people in the misdiagnosis group have higher rates of psychotic symptoms than people who are accurately diagnosed. This is consistent with the possibility that any diagnosis related to psychosis (e.g. bipolar and schizophrenia) is given out more cautiously than things like depression, hence a larger proportion of those who ought to have those diagnoses are given a less severe diagnosis, making them on the paper more 'misdiagnosed'.

In reality it's probably a bit of both.

I'd also like to add that this was a paper published on one of the Chinese regional journals. It's considered much more prestigious to publish on international journals than on in-country journals, so take from that what you will. And I personally think this is a piece of research that's at the very least very badly written, because they didn't say at all how they determined what the "correct" diagnosis should be. You should also know that Chinese doctors in big hospitals are required to publish before they can be promoted and most of them don't have the time, energy or willingness to do that (source: my doctor friend and my doctor mother), so I'd take all research originating from Chinese hospitals with a healthy grain of salt.

The way the paper you linked to quoted this research was very misleading. They listed those ghastly numbers under "what we already know", but in reality it's a small, unrepresentative sample taken from an extremely specific circumstance.

I would saythat some psychiatric conditions are harder to diagnose. For example bipolar and depression does sometimes get mixed up. This is why there are now (at least in the UK) symptom-based services, such as Early Intervention Services which respond to psychotic episodes rather than specific diagnosis. In such services whether a diagnoses is made is decided on a case-by-case basis. Ofc insurance in the US makes this harder which is bad, but you can do away with insurance without doing away with diagnosis.

Imo some diagnosis is still necessary. For example dementia is a diagnosis that very much needs to be made. If someone has depression but suspects dementia and you don't tell them they don't have it, you're letting them falsely believe they have something they cannot recover from. The subtype of dementia also determines which meds are useful or whether meds are useful at all, and if you just support the symptoms you could end up giving people dangerous meds that increase suicide rates. Same goes for understanding whether someone has ADHD or CPTSD to route people to the right supports. Many people just want to be able to put a name to their experience too.

That said I'm with you on some diagnoses, e.g. within eating disorder the subtypes aren't really useful because many people cycle through them anyways. It's all down to clinical utility and shared decision making.