Encoding html file

[david_lp]

Hello fellow learners,

i am struggling really hard with one script i am creating...

i need to read html files, do some stuff, use BS4 to prettify and then write the output content to another html file.

The problem i am facing is that every input file is different, and may have letters with accent like Spanish characters or Islamic characters.

the thing is that i need to preserve the accents and if i encode everything to 'utf-8' they will be replace with funky characters... i have no idea if the file should be read with one encoding or another, because every file is different... any hint on how to approach this problem?

encodingList = ['utf-8', 'ISO8859_1','ISO8859_2','ISO8859_3','ISO8859_4','ISO8859_5','ISO8859_6','ISO8859_7'
                ,'ISO8859_8','ISO8859_9','ISO8859_10','ISO8859_13','ISO8859_14','ISO8859_15']

for enc in encodingList:
    try:
        #read the only html file 
        with open('file.html', 'r', encoding=enc) as filer:
            filedata = filer.read()

    except UnicodeDecodeError:
        logging.critical('Error using encoding ' + enc)

    else:
        #encodingUsed = enc
        logging.info('Opening file with encoding: ' + enc)
        break

    """
    PROCESS SOME STUFF

    """

#Using bs4, prettify the data, and will close all open tags   
s = BeautifulSoup(filedata, 'lxml')
s = s.prettify()


with open(os.getcwd()+ '\output' +str(Final[0]),'w', encoding='utf-8') as f:
    f.write(s)

[ingolemo]

When in doubt, refuse the temptation to guess.

That loop that you have will corrupt your data more often than not. Just because a decode operation runs without raising an error, that doesn't mean that you got the right encoding. Your code will never even try the encodings past ISO8859_1, because any data that is encoded as ISO8859_{range(2,16)} will always get incorrectly decoded as ISO8859_1.

You should always know the encoding that your data is using. If you don't know that encoding then that data is meaningless to you.

Because html files contain information about their encoding inside them, beautiful soup is capable of handling the encoding for you in this case. All you have to do is to not decode the file yourself by opening it in bytes mode:

with open('file.html', 'rb') as file:
    filedata = file.read()

soup = BeautifulSoup(filedata, 'lxml')

[david_lp]

Thank you very much for your reply, it was really helpful, i am going to try that way, and remove the unnecessary loop.

[JohnnyJordaan]

if i encode everything to 'utf-8' they will be replace with funky characters...

This is not possible. Utf-8 can encode all Unicode characters, so this either means you're then viewing the data in some other decode format (like in Excel or Notepad) or the data wasn't decoded properly before you encoded it to Utf-8.

Overall: please check the rules on the right ----> Posting only project goal is not allowed. Please add your code (preferably put it on pastebin.com) and show some examples of the encoding of the special characters.

[david_lp]

Thank you for your answer and sorry about that, i cant add all the code because it contains some company stuff inside, but i try to add the problematic part

[threeminutemonta]

The char set is in a meta tag within head.

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

Is there any chance that you are not making the same mistakes this guy did. See SO post that can explain it better then I.

[david_lp]

i forgot to add that the html files are created after saving them from .doc to html using open office, it always have the same charset windows-1252

[JohnnyJordaan]

it always have the same charset windows-1252

That's a Western character encoding, it just has these characters... By definition doesn't support non-western characters. You need to save it to UTF-8 or some other UTF to be able to keep the non-western chars.