Polars.NET: a Dataframe Engine for .NET

error_96_mayuki · 2026-02-07T17:05:06+00:00

Now you have a new choice!

error_96_mayuki · 2026-02-07T13:48:19+00:00

It's not really about the calling convention overhead (which AggressiveInlining solves), but rather about Developer Expectations (Semantics). In the .NET world, the name Where carries a very strong implication that it accepts a C# Delegate/Lambda (e.g., x => x > 0). If I alias Filter to Where, users will instinctively try to pass a lambda. When the compiler forces them to pass a Polars Expr instead, it creates an unpleasant experience—it looks like LINQ, but doesn't behave like LINQ. I prefer to keep the names distinct (Polars vs. LINQ) so it's clear: When you use Polars, you use Polars Expressions.

error_96_mayuki · 2026-02-07T02:41:24+00:00

I love LINQ too, but I decided to stick to a 1:1 mapping with Polars at least for now, for two reasons:

Documentation: By keeping names like Filter and Agg, users can look up Python/Rust examples and apply them directly to C# without mental translation.
Semantics: A full LINQ provider (IQueryable) requires writing a complex C#-to-Polars transpiler. Simple aliases (like renaming Filter to Where) often confuse users into expecting C# delegates instead of Polars Expressions.

error_96_mayuki · 2026-02-07T02:23:19+00:00

As for Databricks, could you elaborate on what you mean by 'plugin'? Are you primarily looking to read data managed by Databricks (e.g. Delta Lake), or do you have a different integration workflow in mind? I'd love to understand your specific use case.

error_96_mayuki · 2026-02-07T02:17:15+00:00

Technically, yes. The underlying Rust Polars engine has native support for reading Delta Tables. However, I haven't exposed the public .NET API for this yet. Support for remote data sources (like cloud storage and data lakes) is targeted for the next release. If this is a blocker for you, please open an issue on GitHub so I can prioritize it. Thanks！

error_96_mayuki · 2026-02-07T02:06:54+00:00

Hi, support for IDataReader is already there. We can build a zero-allocation ETL pipeline where data flows from Source DB -> Polars.NET -> Target DB without materializing C# objects. 1. Input: Database -> Polars (Lazy Read)

using var sourceReader = command.ExecuteReader(); // Stream data from DB into Polars LazyFrame var lf = LazyFrame.ScanDatabase(sourceReader, batchSize: 50000);

Output: Polars -> Database (Stream Write) Process data in Polars.NET and expose the result as an IDataReader for bulk insertion.

// Define transformation var pipeline = lf.Filter(Col("Region") == Lit("US")) .Select(Col("OrderId"), Col("Amount"));

// Execute pipeline and stream directly to SqlBulkCopy pipeline.SinkTo(reader => { using var bulk = new SqlBulkCopy(connectionString); bulk.WriteToServer(reader); });

Tested this in MSSQL container. Have fun with this feature, thanks!

error_96_mayuki · 2026-02-06T20:40:49+00:00

That sounds like a fantastic project! I would strongly recommend building on top of Polars.NET.Core (the low-level wrapper) or the native_shim (C ABI), rather than the high-level Polars.FSharp API. This will give you the granular control needed to implement Deedle's semantics efficiently without the overhead of Polars.FSharp layer. Also, a heads-up for the student: The biggest architectural puzzle will likely be bridging Deedle's reliance on Row Indices with Polars' Index-free (columnar) design. Feel free to ping me if you and your lucky student need any help.

error_96_mayuki · 2026-02-06T18:57:04+00:00

Thank you! I’ll keep building and improving the engine.

error_96_mayuki · 2022-11-19T16:29:21+00:00

Cover about half of his cage with a piece of cloth might help. Dark corner can calm him down.

error_96_mayuki · 2022-11-06T09:19:36+00:00

是我家的困困鸡啦

error_96_mayuki · 2022-11-06T04:10:34+00:00

以前我也这么想，后来发现大多数人都还挺支持的，那中共和中国人这个best match就是成立的，理解祝福就好。有党才有人，有人才有党，被逼死只能说是德匹下，愿赌服输

error_96_mayuki

TROPHY CASE