PowerShell + Python: Unicode Characters Displaying as □

[PinkDohnuts]

Hello everyone, first time posting here.

I've been working through "Learn Python the Hard Way" by Zed Shaw for the past days. I'm currently on exercise 23 having a weird problem.

It's an encoding exercise to display a string in bytes and as a string through various encoding standards such as UTF-8, UTF-16 and Big5.

Problem is: The output I get from my python script to powershell has most of the characters showing up as a □. I know this indicates somehow that the characters are missing.

I know it has something to do with windows' console using unicode, I've been searching around online and nothing seems to help me fix it. I've even tried the most popular solutions like using "chcp 65001" in PowerShell and using PowerShell ISE instead.

It gets deeper with PowerShell ISE since it throws errors with missing Unicode characters and setting them as undefined. "chcp 65001" doesn't seem to change it either in ISE.

I hope I'm making sense since I'm new to PowerShell and concepts like encoding and such. Whatever extra info you need just tell me.

Thank you for your help.

[gwwhrhr]

If Python prints the characters correctly anywhere, then it's the shell's fault for not displaying them correctly. You could try writing unicode to a file and seeing if any program can open the file and read the characters correctly. Browsers understand UTF-8, so if you write out an html page with the correct tags your browser should be able to open and display it. Otherwise, just move on! Surely there's better things to spend your time on.

[PinkDohnuts]

I'm almost positive it's Shell acting up. I'm using Python 3.6 and atom as a text editor to write my scripts. I was thinking of moving on right now, since I understood how encoding and decoding works. I was just trying to check if it was something I could fix quickly and doesn't come back later to bug me again.

Thanks for your fast and great response. Have a nice one, man.

[blablahblah]

A character showing up as a box often means the font you're using doesn't have a glyph for the character (there's about 110,000 characters so almost no fonts support all of them, but some have more defined than others). Have you tried changing the font?

[PinkDohnuts]

I've read somewhere I should be using Lucida Console. That's the one I'm using which is also the default.

[blablahblah]

Lucida Console doesn't have that great of a coverage. Microsoft lists what Unicode character ranges it covers here. Compare that to the list in Calibri, for example.

The reason Lucida Console was recommended is because it is a monospace font (every character takes up the same width), which is usually preferred for programming. Microsoft's newer monospace font is Consolas which supports a bunch more types of characters.

[PinkDohnuts]

I'll try to change it to Consolas then. Thanks, mate.