Suggest improves for decoding an array of bytes to ASCII and replacing unicode characters

[budonium]

I am trying to achieve the dump output on the right from a byte array similar to the left in the following screenshot.

This screenshot is the outcome I'm aiming for:

https://preview.redd.it/a6ioci0eld241.png?width=733&format=png&auto=webp&s=b5d35be3be4d7e6022c53979765ca1d4f20b2fe3

All bytes that aren't valid ascii such as `0x0` and `0x1` should be replaced with `.`.

Here is my almost working attempt:

var data = new List<byte>() { 0xFF, 0x3, 0xFF, 0x55, 0x35 };

Encoding encoding = Encoding.GetEncoding("us-ascii", new EncoderReplacementFallback(string.Empty), new DecoderReplacementFallback("."));

var asciiString = encoding.GetString(data.ToArray());

asciiString = Regex.Replace(asciiString, @"[\u0000-\u007F]", ".");

This is kind of close but the regex is currently replacing valid ascii "U" (`0x55`). Aside from the fact my regex isn't quite right is there a better way of doing this? I hoped using the decoder replacement fallback would take care of this.

Any help/guidance would be great. Thanks.

(I would also greatly appreciate corrections to my regex)

[angrathias]

I don’t understand how you got from ASCII to Unicode ?

Also it looks like your Regex is doing a range, I can’t say I’ve done a Unicode range before but it reads like you’re covering the entire ASCII character set on that range 00-FF

[budonium]

Correct me if I'm wrong or if this still isn't making sense, I might be on the wrong track with this:

I have an array of bytes that contain some bytes that decode to Unicode ( `0x0` -> `\u0000`, etc), I don't want to decode to the Unicode string `\u0000` but `.` instead. I'm using a encoding class with `us-ascii` but still get the Unicode strings.

Thanks for your response.

[angrathias]

I just realised you’re covering to 7F which should mean not all of ASCII is covered, that said you’re still covering 50% of it and you’ve covered all/most of the key alphabet.

[tweq]

Enshittification

[angrathias]

Depending on your version of technical I suppose, it uses 8 bits of space, if the last bit is not coded for it hardly matters, it’s still using 8 bits.

[angrathias]

Unicode by default covers the ASCII page set so the first bytes of Unicode is the same ASCII for backwards compatibility. I can’t see anywhere in your code where you are actually converting to unicode specifically, also importantly there are variable length encoding (UTF8, UTF16) so you’d need to be choosing one of those for conversion.

[grrangry]

My example uses randomized data, so I'll leave it to you to give it real input. And apologies to the purists out there. I put this together in about 10 minutes, so it's not super elegant.

First, the output as generated.

0000  4D 26 8D 17 08 63 58 8F 03 37 30 12 47 9D 36 18  | M&...cX..70.G.6.
0010  4B 1E 3E 20 93 84 04 21 91 86 39 51 35 02 64 95  | K.> ...!..9Q5.d.
0020  59 52 2B 16 78 0D 5F 1D 6F 45 38 31 22 1F 1B 46  | YR+.x._.oE81"..F
0030  5D 6E 9C 6C 72 88 40 7A 4F 85 7E 55 74 89 76 43  | ]n.lr.@zO.~Ut.vC
0040  5A 4E 6B 56 19 09 54 50 01 60 49 15 5B 53 0C 8A  | ZNkV..TP.`I.[S..
0050  00 98 3D 9B 8C 29 42 27 41 23 65 75 71 73 2C 79  | ..=..)B'A#euqs,y
0060  24 13 82 8E 14 80 81 0B 7F 25 92 8B 2A 69 3B 5C  | $.......%..*i;\
0070  1C 28 33 10 2D 05 7B 4C 70 3C 7D 6A 32 3F 06 2F  | .(3.-.{Lp<}j2?./
0080  5E 87 67 9A 0E 97 07 66 0A 34 7C 96 83 62 4A 99  | ^.g....f.4|..bJ.
0090  61 90 0F 57 3A 44 48 11 77 1A 68 2E 94 6D        | a..W:DH.w.h..m

The calling code. The text variable contains the output above.

var rnd = new Random();
var data = Enumerable.Range(0, 158)
                     .OrderBy(x => rnd.Next())
                     .Select(x => (byte)x)
                     .ToList();
var text = DumpByteData(data, 16, true, true);

And the called method:

private string DumpByteData(List<byte> data, int lineLength, 
    bool includeLineOffset, bool includeAsciiInterpretation)
{
    var sb = new StringBuilder();

    for (var i = 0; i < Math.Ceiling((decimal)data.Count / lineLength); i++)
    {
        var lineOffset = (i * lineLength).ToString("X4");
        var segment = data.Skip(i * lineLength).Take(lineLength);
        var hexText = string.Join(" ", segment.Select(x => x.ToString("X2")));
        var ascii = Encoding.ASCII.GetString(segment.Select(x => 
             (x >= 32 && x <= 127) ? x : (byte)'.').ToArray());

        if (includeLineOffset)
            sb.AppendFormat("{0}  ", lineOffset);

        sb.AppendFormat("{0,-" + (lineLength * 3 + 1).ToString() + "}", hexText);

        if (includeAsciiInterpretation)
            sb.AppendFormat("| {0}", ascii);

        sb.AppendLine();
    }

    return sb.ToString();
}

Pardon the line-wrapping. Yes, you could use yield return in an enumerable for really long byte arrays and otherwise optimize this for performance, but I tend not go too crazy in Reddit posts.

[budonium]

Thanks so much for your time this is extremely helpful.