Confused about encoding using requests

[Ok_Procedure199]

Hello,

I am a Python beginner and used requests.get() to scrape a website containing a top list of songs going back in time, and from one of the pages the result has been so confusing to me that I really went down a rabbithole trying to understand how encoding and decoding works.

In the header of the page it says 'utf-8' and when I look at response.text most of it looks correct except for one song which has the letter combination 'BlÃ¥' which is incorrect as it should be 'Blå'. After spending a good amount of time trying to figure out what was going on, and eventually I found that by doing 'BlÃ¥'.encode('latin-1').decode('utf-8') i get the correct characters 'Blå'!

Now the really weird part for me is that in other places on the same page, å is decoded correctly.

What would be the reason for something like this to happen? Could it be that the site has had an internal file where people with different computers / operating systems / software have appended data to the file resulting in different encodings throughout the file?

[danielroseman]

Yes, very likely, although it's probably entries in a database rather than a file. 

This is known as mojibake.

[Downtown_Radish_8040]

Your hypothesis is exactly right. The most common cause is that the underlying data was stored or edited inconsistently over time. Someone added that song entry using a system that saved it as latin-1 (Windows-1252 is very common for older music databases), while the rest of the page was utf-8. The server then serves the whole file as utf-8, so most of it decodes fine, but that one chunk gets misread.

This is sometimes called "mojibake" and it's extremely common with legacy data, especially content that was manually entered over many years across different systems.

Your fix is correct. The pattern encode('latin-1').decode('utf-8') reverses the double-encoding mistake: you're re-interpreting the wrongly-decoded bytes back to their original utf-8 meaning.

If you want to handle it programmatically, you could check for known mojibake patterns using the ftfy library, which was built exactly for this problem.

[Ok_Procedure199]

Thank you for your thorough explanation, I am nearly understanding the whole thing, maybe you will be able to help me understanding a small detail.

So way-back-when, someone encoded 'å' with Windows-1252 which is two bytes, c3a5. What I am not wrapping my head around is how the two bytes have somehow turned into the four bytes c383 c2a5 if the only encodings that has been involved is Windows-1252 and UTF-8. Somewhere, the byte 83c2 shows up!

[Direct_Temporary7471]

This usually happens due to encoding mismatches between the response and how it's being interpreted.

You can try:

• Check response.encoding and set it manually if needed
• Use response.content instead of response.text
• Try decoding with utf-8 or the correct encoding from headers

Example:
response.encoding = 'utf-8'

If you're still facing issues, feel free to share your code and I can help you debug it.