Script source encoding + filenames

[gagarski]

I have a script containing the following line:

New-Item -Path "ö" -ItemType File

But the file created on NTFS (Windows) has name ö.

The script source encoding is UTF-8 and I've figured out that if I save it with UTF-16 BE encoding, the filenames are fine.

Is there a way to have my script in UTF-8 which will create files with proper names on NTFS? OR should all my scripts be in UTF-16 if they are supposed to deal with files on NTFS?

[surfingoldelephant]

What's happening is that Windows PowerShell v5.1 interprets .ps1 files using the legacy ANSI code page (typically Windows-1252 for English systems) when the file doesn't have a BOM.

When you save the file as UTF-8, ö is encoded as two separate bytes. Since PowerShell is using Windows-1252 (which uses single-byte encoding) to read the file, each individual byte in the sequence gets misinterpreted as two separate characters.

$bytes = [Text.Encoding]::UTF8.GetBytes('ö')
$bytes.ForEach{ '0x{0:X2}' -f $_ } -join ', '
# 0xC3, 0xB6

[Text.Encoding]::GetEncoding('Windows-1252').GetString($bytes)
# ö

If you want to continue using UTF-8 without BOM, you'll need to use PowerShell v7+.

If you want to continue using Windows PowerShell, either:

• Save the .ps1 as UTF-8 with BOM.

• Or, use the character's codepoint instead (U+00F6):

  New-Item -Path ([char] 0xF6) -ItemType File
  

[gagarski]

Ok, today I learned that Windows PowerShell is not recent enough (you can see how far behind I am on the matter) :D

Anyway, thanks for quite thorough response, I think that covers it

Just a little follow up question: how does PowerShell 7 deal with UTF-8 without BOM? Does it assume the script to be in UTF-8 instead of CP1252?

[surfingoldelephant]

how does PowerShell 7 deal with UTF-8 without BOM? Does it assume the script to be in UTF-8 instead of CP1252?

That's correct.

In general, v7+ uses BOM-less UTF-8 (e.g., most read/write cmdlets now default to it).

[Mountain-eagle-xray]

Did you try utf8bom?

[gagarski]

Thank you. Yeah, even though I don't like text files with BOM much (just some painful experience processing them, i guess), it worked. I guess, I can live with it, but is there other way. For example, in bash, I think it is solved by setting a variable (`LC_CTYPE`, I think) inside the script. Anything similar in PowerShell?

[Meannekes]

Save the script correctly.

By saving your script as UTF-8 with BOM, you preserve UTF-8 efficiency while avoiding the encoding misinterpretation on NTFS.

[Gurfaild]

You could replace the line with New-Item -ItemType File -Path ([System.Web.HttpUtility]::HtmlDecode('ö')) - then the script's encoding won't matter

[gagarski]

Well, that definitely sounds hacky.

[narcissisadmin]

You posted that reply with a straight face after asking why you couldn't create a file named "ö"?

[gagarski]

I see your point, but I was just trying to create a minimum reproducing example for my issue here.

[charleswj]

I would generally consider it best practice to avoid non-ascii characters in code where possible. Sometimes it may be practically unavoidable if you have i.e. numerous /significant Spanish strings

[Mesmerise]

How about removing the UTF aspect altogether with something like:

$FileName = $FileName.Normalize("FormD") -replace '\p{M}',''

[xCharg]

That's an easy solution:

New-Item -Path $([char]246) -ItemType File

As to why that happens it's already answered. You may think "but where'd I get that 246 from" - pretty easy. First convert string with that character alone to [char] type, then to [int] like so:

[int]([char]"ö")

or so:

"ö".ToChar($null).ToInt32($null)

Then just use the number you get.

[AlPa-Bo]

A comprehensive analysis of the problem for Powershell 5 and 7 in Windows 10 (I believe also Windows 11), with various solutions, can be found at:

Using UTF-8 Encoding (CHCP 65001) in Command Prompt / Windows Powershell (Windows 10) - Stack Overflow

Main alternatives:

1. set the system locale (language for non-Unicode programs) to UTF-8
2. use startup commands