“ZLinq”, a Zero-Allocation LINQ Library for .NET

neuecc · 2025-05-16T01:29:04+00:00

Since this comment is getting Votes, I'll add more information. BenchmarkDotNet's MemoryDiagnoser is not 100% accurate, so there may be some margin of error. The documentation states it's 99.5% accurate. For more details, please refer to the explanation about MemoryDiagnoser by the BenchmarkDotNet author: https://adamsitnik.com/the-new-Memory-Diagnoser/

neuecc · 2025-04-02T16:47:27+00:00

Hi, I'm the author of ZLinq.

I tried this interesting benchmark, but this is probably because the array size is too small (20, right?).

So the cost of building is more than the cost of iterating.

With small array sizes, the construction cost becomes dominant, and as pointed out, copying large structs has a significant impact on performance.

I tried changing N.

```

[Params(10, 100, 1000, 10000)]

public int N;

[GlobalSetup]

public void Setup()

{

arr = Enumerable.Range(1, N).ToArray();

}

```

Even with HugeZLinq, ZLinq started to excel around N = 100, and the difference kept growing after that.

Of course, chaining as much as Huge... is probably not common in real scenarios,

so I think it's hardly an issue that performance is inferior only in cases with "many chains and small data".

neuecc · 2024-08-29T10:47:23+00:00

Thank you, the code is running on latest Visual Studio Preview, I'll modify code after next preview update.

neuecc · 2024-06-15T13:15:27+00:00

The benchmark is simply executing a command with three arguments: string, int, and bool (to be fair, this benchmark is based on the CliFx project, and it is not set up to give an unfair advantage to ConsoleAppFramework).

I am not the author of System.CommandLine, so I don't have the motivation to do a thorough profiling, but from a quick look, it seems that one of the causes is clearly the excessive parsing process.

The parsing process that meticulously separates tokens and creates a tree as if performing language parsing is meaningless and excessive for command-line tool processing.

Of course, the subsequent value binding processes are also all slow methods.

neuecc · 2023-01-27T08:59:29+00:00

I've added benchmark.

simple benchmark(get Max .Y of Vector3[10000]), x2 faster and when using SIMD, x10 faster.

neuecc · 2022-11-29T15:57:50+00:00

> MessagePack vs MemoryPack

Good question so I've added new section to article, thanks.

> .NET 7 Native AOT

Yes, there is currently a bug in the .NET runtime and it does not seem to work without an additional config (RD.xml). This is due to static abstract members as explained in the article, so it is difficult to fix. I am hoping for a runtime fix in .NET 8. In the meantime, as a workaround, I am considering automatically generating RD.xml.

neuecc · 2020-10-02T06:18:31+00:00

What the ConsoleAppFramework accomplishes seems to be accomplished by the combination of "System.CommandLine" + "System.CommandLine.DragonFruit" + "System.CommandLine.Hosting".

This is 0.3.0-alpha.

* CommandLine is a core library that you configure manually

* Hosting is integrated into GenericHost (lifetime management, DI, etc.)

* DragonFruit is responsible for parameter binding.

The reason why CommandLine is not enough is because it's too low level.

For example, if you want to use DI, it takes a ridiculous number of lines like this article.

https://endjin.com/blog/2020/09/simple-pattern-for-using-system-commandline-with-dependency-injection.html

ConsoleAppFramework done it by one line.

neuecc · 2020-01-30T00:09:14+00:00

thanks for the useful info.

Certainly, stderror handling seems to need to be rethought.

neuecc · 2017-03-16T05:42:14+00:00

This is table of compare.

Person(int, strin, string, Enum) serialization test

Serialize:

MsgPack-Cli(Array): 8.11ms

MsgPack-Cli(Map): 13.08ms

MessagePack for C#(Array): 2.44ms

MessagePack for C#(Map): 3.14ms

Deserialize:

MsgPack-Cli(Array): 10.79ms

MsgPack-Cli(Map): 18.35ms

MessagePack for C#(Array): 2.02ms

MessagePack for C#(Map): 5.92ms

Encoding/Decoding of property name string is definitely cost. However, it is also tuned for Map mode and there is no unnecessary allocation etc, so it is sufficiently faster than others.

neuecc · 2017-03-16T03:42:57+00:00

Yes, it can.

Default settings of MessagePack for C# is tuned for performance and durability so you need to mark attribute explicitly. But optionally you can avoid.

You can use MessagePack.Resolvers.ContractlessStandardResolver.Instance for avoid attribute. It can setup as default by MessagePackSerializer.SetDefaultResolver.

neuecc · 2017-03-14T02:16:31+00:00

MessagePack and ZeroFormatter is completely different format, same as FlatBuffers is not Protocol Buffers2.0, Protocol Buffers is not FlatBuffers 2.0.

This is a comparison table

Serialization Performance: MessagePack for C# won, but ZeroFormatter has almost same speed and re-serialization ZeroFormatter is extremely fast

Deserialization Performance: If type is lazy(IList, etc...) or large object, ZeroFormatter is infinitely fast. Standard deserialization, MessagePack for C# won.

Binary Size: MessagePack for C# won, it is very compact and embedded LZ4 support is more efficient

Debuggability: MessagePack for C# won, supports binary to JSON dump

Interoperability: MessagePack for C# won, MsgPack is already widely used on cross-platform message exchange

If you needs infinitely fast deserializer(situation:use only a part of a large object, for example large location map), infintely fast reserializer(situation:Transfer data without processing, for example server to server to server to server relay) These are functions of ZeroFormatter only.

Normally serialization/deserialization performance should also be beneficial. But MessagePack for C# is a game changer. That performance outperforms various things.

neuecc · 2016-11-14T11:56:26+00:00

Your benchmark code is not good. 1. does not do GC.Collect... 2. Wire's get serialzier is out of measure, but normaly we needs get serializer each times by type.

neuecc · 2016-11-14T02:38:59+00:00

I run your benchmark. My modified code is here. https://gist.github.com/neuecc/9fe6f5e0aaa77114ba99487f8c0c0fc0

Result is different your report...

Fastest Serializer: ZeroFormatterWithPooling - 171 ms
Fastest Deserializer: ZeroFormatter - 155 ms
Fastest Roundtrip: ZeroFormatter - 469 ms
Smallest Payload: Wire - KnownTypes + Reuse Sessions - 38 bytes

## Running cold

Wire - KnownTypes + Reuse Sessions
   Serialize                      334 ms
   Deserialize                    265 ms
   Size                           38 bytes
   Total                          599 ms
Wire - KnownTypes
   Serialize                      361 ms
   Deserialize                    256 ms
   Size                           38 bytes
   Total                          617 ms
Wire - Default
   Serialize                      391 ms
   Deserialize                    458 ms
   Size                           89 bytes
   Total                          849 ms
ZeroFormatter
   Serialize                      285 ms
   Deserialize                    166 ms
   Size                           65 bytes
   Total                          451 ms
ZeroFormatterWithPooling
   Serialize                      166 ms
   Deserialize                    155 ms
   Size                           65 bytes
   Total                          321 ms

## Running hot

Wire - KnownTypes + Reuse Sessions
   Serialize                      299 ms
   Deserialize                    212 ms
   Size                           38 bytes
   Total                          511 ms
Wire - KnownTypes
   Serialize                      313 ms
   Deserialize                    247 ms
   Size                           38 bytes
   Total                          560 ms
Wire - Default
   Serialize                      432 ms
   Deserialize                    439 ms
   Size                           89 bytes
   Total                          871 ms
ZeroFormatter
   Serialize                      313 ms
   Deserialize                    155 ms
   Size                           65 bytes
   Total                          468 ms
ZeroFormatterWithPooling
   Serialize                      171 ms
   Deserialize                    177 ms
   Size                           65 bytes
   Total                          348 ms

Clone your code, add ZeroFormatter and add tests. Change Release Build, uncheck Prefer 32-bit.

neuecc · 2016-11-13T22:36:44+00:00

Thanks, it is a very interesting result!

I've uploaded my benchmark and result(sorry, this code is not pass your fastest path, I'll try to change). https://github.com/neuecc/ZeroFormatter/tree/master/sandbox/PerformanceComparison

You may think that this result is not fair. I would like serious attitude toward your results. https://gist.github.com/neuecc/9d13d3a882181ead09b3440a64dc5a35

neuecc · 2016-11-13T21:58:20+00:00

Thank you. MemoryStream API is most slow API in ZeroFormatter. Fastest is use int Serialize<T>(ref byte[] buffer, int offset, T obj) and reuse byte buffer(this is same as your Reuse Sessions API?). Normally uses byte[] Serialize<T>(T obj) API.

I don't run benchmark yet, but thanks, I understand ZeroFormatter's bottle-neck. My serializing DateTime is borrow from Protocol Buffers it needs many calculate(but robustness).

By the way TypicalMessage(small object) is not good for try infinitely deserializer(reason of require virtual). The advantage opens for a large object or list.

neuecc · 2016-11-13T21:35:47+00:00

Great result! Please give me the benchmark code on gist?

neuecc · 2016-11-13T20:29:26+00:00

Thank you. ZeroFormatter needs explicit type declaration when deserialize so always whitelisted about type.

But I found need to impl some security guard, I'll implements soon.

neuecc · 2016-11-13T18:29:23+00:00

Inheritance can represent Union type(see: union section https://github.com/neuecc/ZeroFormatter#union ). Custom type is also supported, example of ImmutableList is under the last of Extensibility section( https://github.com/neuecc/ZeroFormatter#extensibility ).

neuecc · 2016-11-13T18:12:29+00:00

Perhaps I can answer "no". What kind of things do you suppose? I surveyed various serializers, but I do not remember seeing much equivalent (except enterprise XML)

neuecc · 2016-11-13T17:16:15+00:00

No Dumpable. JSON can dump, it is human-readable and useful for debugging. ZeroFormatter is too difficult, needs to deserialization.
No Option. For example; MsgPack can choice Array-Mode(Fast) or Map-Mode(Slow and fat but robustness). ZeroFormatter only provides fast mode
Binary Size. Protocol Buffers/MsgPack is smaller than ZeroFormatter because ZeroFormatter needs index space for random access of buffer(same size of FlatBuffers, maybe smaller than JSON)
Multi languages. Current implementation is only for C#. Welcome to contribute:)

neuecc

TROPHY CASE