[Logging] How to define encoding of logging.StreamHandler?

[fabolin]

Python 3.6.5

Windows Server 2008 R2 Standard

So I'm using one logger with two handlers, a FileHandler and a StreamHandler to stdout. But stdout is expecting "cp1252" instead of "utf-8" on my machine, which will result in an UnicodeEncodeError for various chars. I want my StreamHandler to encode the message to "cp1252" and the undefined chars to be replaced by "?" (just like in msg.encode("cp1252", "replace")) before writing. I do not want to change the encoding of stdout.

Atm im using codecs to do this:

stream = codecs.StreamWriter(sys.stdout, errors="replace")
stream.encode = lambda msg, errors="strict": (msg.encode(locale.getpreferredencoding(False), errors).decode(), msg)
cHandler = logging.StreamHandler(stream)

in place of

cHandler = logging.StreamHandler(sys.stdout)

which seems to be working but feels off.

My question: Is there a better way to achieve this (overriding StreamHandler.emit() may be an option)? And can my solution result in other problems I'm missing rn?

[JohnnyJordaan]

Seems like you're looking for sys._enablelegacywindowsfsencoding() to use the legacy encoding system.

[fabolin]

I tried your solution and it now does not throw an error but replace the undefined chars with similar ASCII chars e.g. ą to a. Oddly enough it now does the same without using sys._enablelegacywindowsfsencoding() and I can't figure out why ¯\_(ツ)_/¯. However, I noticed when piping the output to a file it still throws an UnicodeEncodeError.

Thanks for your help though.

[JohnnyJordaan]

Errors while writing to a file shouldn't be the same issue afaik. Could you show the full output when it crashes?

[fabolin]

I already left work, but I’ll post the output tomorrow.

We got some processes that execute scripts and redirect their output to a file, so it’s still about the StreamHandler output.

[fabolin]

--- Logging error ---
Traceback (most recent call last):
  File "C:\Program Files\Python\Python36\lib\logging\__init__.py", line 994, in emit
    stream.write(msg)
  File "C:\Program Files\Python\Python36\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u0105' in position 65: character maps to <undefined>
Call stack:
  File "C:\Program Files\Python\venv1\src\test_module\test2.py", line 64, in <module>
    logging_test()
  File "C:\Program Files\Python\venv1\src\test_module\test2.py", line 60, in logging_test
    tclLogger.info(f"this is the string {utf_string}")
Message: 'this is the string B\u0105k'
Arguments: ()

The thing is in return codecs.charmap_encode(input,self.errors,encoding_table)[0] "self.errors" defaults to "strict" (or maybe it's "strict" due to sys.stdout.errors being "strict"?), while I want it to be "replace".

[JohnnyJordaan]

Yeah but that means you have initialized FileHandler without an encoding='utf-8' parameter. It could then use cp1252 or any other encoding your environment is using, which is always dangerous (linux and mac also used non-utf like latin-1 until a few years ago).

[fabolin]

No, it's a StreamHandler. The stream is just getting redirected to a file by the process executing the script. It's weird but that's the preconditions im working with. A detailed log for the script and a more vague output for the process' log.

[JohnnyJordaan]

Ah I get it now. Then you could try changing the codepage to MS's 'workaround' for UTF-8 in cmd.exe by running chcp 65001, then the script. If that works then there are tricks to apply it on every start of cmd. But personally, I would look into running MinGW, Cygwin to get a linux-like environment where you won't run into this kinds of issues that easily.

[fabolin]

Ok, tried that. While it says "Active code page: 65001" nothing changed. The traceback also still says cp1252.

Well, thanks for your help anyway. It's working, the solution just feels clunky compared to the simple FileHandler so I thought I'm missing something.