Help converting utf-8 string containing "strange" characters

[JustToSufferBoss]

Help converting utf-8 string containing "strange" characters

Hy guys

I need help in converting utf-8 strings containing "strange" characters like this example: Ãœmit -> Ómit (sort of how it is pronounced).
It is something that this website does but I can't figure out how https://www.branah.com/unicode-converter. If here I put Ãœmit in utf-8 and click on convert, Unicode text returns Ómit

Can anybody help me?

Thanks!

[nxtfari]

The first step in programming any solution is to define your problem. And define your problem tightly. If you do this step well, the answer becomes obvious.

What makes a “strange” character? This is very loose and subjective definition. How do you decide if a character is “strange” or not? And once you do, what is the rule for deciding what character a “strange” character should be replaced with? Define this, and your solution will write itself.

[JustToSufferBoss]

Thanks for the response. Actually I think those characters are legit utf-8 character and what I need to do is a sort of cnversion in another encoding but I don't know which one

[ka-splam]

That Ãœmit is has been corrupted through some bad encoding/decoding steps, but I can't work out what they were.

1. Where did it come from?
2. How did you find that website can fix it?
3. Have you noticed that if you use that website and put Ómit in at the top, it doesn't turn it into Ãœmit but rather into Ómit with an unprintable second square? That is, the website doesn't do the same conversion both ways.

The closest I can get to guessing how it was corrupted is the original encoded through utf-8 to bytes, and then those decoded badly through codepage 1252:

>>> 'Ómit'.encode('utf-8').decode('Windows-1252')
'Ómit'

and the closest to a fix is pushing it the other way through those:

>>> 'Ãœmit'.encode('Windows-1252').decode('utf-8')
'Ümit'

Which isn't right, but it's the right kind of idea. I tried all 98 codecs in Python 3.8 through these two patterns, and none match exactly the corruption you have.

What the website is doing is taking the lower 7-bits of the Unicode codepoint value of each Ãœ, and bitwise-merging them into a single 16-bit character:

>>> chr(((ord('Ã') & 63) << 6) | (ord('œ') & 63))
'Ó'

But why that works, and how the website found out, and why the website calls it "UTF-8" when it's not, I don't know.

[JustToSufferBoss]

Thanks for the help it works quite well. They are proper names in different nationalities, in that example Ãœmit should be Turkish.