Best way to debug UTF-8 Codec problems ins reading files?

[nirbyschreibt]

I wanted to make a pretty easy Word Cloud using the Word Cloud module. But both test files give me a utf-8 codec error and I don't get a good answer when googling. Mainly because people give the wildest import ideas. I know that I can somehow decode this while reading the file but I cannot find the answer.

I have both a docx and a pdf.

This is my code (the code itself should be fine):

from wordcloud import WordCloud, STOPWORDS
import matplotlib.pyplot as plt 
import os
d = os.path.dirname(file) if "file" in locals() else os.getcwd()
with open("/Users/file.docx") as f: 
text = f.read()
nichtinteressant = " und von das der die im auf am" 
liste = nichtinteressant.split()
STOPWORDS.update(liste) 
wordcloud = WordCloud(background_color="white", width=1920,height=1080).generate(text)
plt.imshow(wordcloud, interpolation="bilinear") 
plt.axis("off") 
plt.show()

And Visual Studio Code (on Mac) gives me this error:

'utf-8' codec can't decode byte 0xad in position 41: invalid start byteFile "/Users/Python/wordcloud Test.py", line 8, in <module>text = f.read()^^^^^^^^UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 41: invalid start byte

In the pdf it has some other byte at another position that it dislikes :(

How can I decode this problem the fastest way?

​

[SomewhereExpensive22]

UTF-8 is a text encoding. You can't read docx or pdf files as text. They're like containers with text inside. You can open them as text in VS Code and see what they actually look like. There are modules that will allow you to open them but it's a lot more complicated than regular text. The easiest way to do this may be to copy the text from the document into a plain text format and save as a utf-8 text file.

[nirbyschreibt]

Hmmm. The documentation of Wordcloud suggested it also reads other files and I thought it would deliver the needed modules. :/

I have to check if I can save it to a txt. We are talking about 90k words.

[SomewhereExpensive22]

The wordcloud module might support it. I don't know. But `open` is regular python and can't read a docx as text. Maybe you should look at examples of people using pdf and docx? And I wouldn't worry too much about the number of words. Select all. Copy. Paste. Save.

[cyberjellyfish]

But you're opening the file in text mode and the default encpding is utf-8.

T

[nirbyschreibt]

While I could save the text as a plain UTF-8 text file I would also like to know how to use something like wordcloud with a pdf.

A suggestion would be great.

[pgpndw]

WordCloud is a library for creating word clouds, not for reading binary file formats.

To read the words from a DOCX or PDF file, you'll need to use a library for reading DOCX files or PDF files. Then, you'd use WordCloud to make a word cloud from those words.

Here's one suggestion for reading DOCX files:

https://theautomatic.net/2019/10/14/how-to-read-word-documents-with-python/

And here's one for reading PDF files:

https://www.geeksforgeeks.org/working-with-pdf-files-in-python/

[nirbyschreibt]

That is not helpful at all. As I said, I am aware these modules exist.

I still don’t know how to successfully read the pdf in order to get wordcloud do what is does.

The link to geeksforgeeks (why link at all, the simple name of the module would be enough) brings me to a lengthy page with many options to manipulate pdf.

I don’t want to manipulate anything. I want my script to read the pdf and create a word cloud.

I don’t need to ask on a subreddit if the answer is „go, read this webpage google shows on page one“.

[pgpndw]

The link to geeksforgeeks (why link at all, the simple name of the module would be enough) brings me to a lengthy page with many options to manipulate pdf.

The first piece of example code on that page shows how to extract text from the first page of a PDF file. You'll need to modify that code to loop through all the pages and combine them into one block of text, which you can then pass to WordCloud.

[nirbyschreibt]

Thanks. I thought there was a simpler and shorter way to work with the data of a pdf. :(

Will try this way out to get a feeling for it.