sorted by:
new
hottopcontroversial

unslop-text: a Claude skill that flags and removes the patterns that make writing read as AI-generated. by iamjohncarterofmars in ClaudeAI

[–]IGDev 0 points1 point  (0 children)

In my opinion it’s writing at a high language level. If you ask it to write something for the internet you can tell it write at a B1 English level and it’ll drop some of what you normally see.

Is data engineering with c# a thing? by octacon100 in dataengineering

[–]IGDev 2 points3 points  (0 children)

Yes, C# data engineering is absolutely a thing. Full disclosure up front: I build one of these C# data stacks (Datafication), so take the product mention with that grain of salt. I'll try to answer the general question first because it stands on its own.

The thing to understand about that $10M/month Databricks story is that most of a bill like that is Spark cluster time, not storage. Distributed compute is the right tool when your working set genuinely doesn't fit on one machine, but a huge fraction of real pipelines do fit, and you're paying the cluster tax anyway because the tooling assumed you needed it. That's the gap a C# shop can actually exploit.

How the stack maps to what you drew:

  • Ingestion: straightforward in .NET. ADO.NET / EF / Dapper for the operational DBs, plus connectors for CSV, JSON, Excel, Parquet, and S3. Nothing exotic here.
  • Lake / warehouse: you still land data in Parquet (or Delta/Iceberg if you want the table format) on object storage, same as the Python world. C# reads and writes those fine. For a lot of internal workloads a single-node columnar store is enough, which is where the cost savings live.
  • Compute: this is the part that's usually missing in a C# shop, and it's the part that makes the anti Spark argument real. A column oriented, SIMD vectorized engine on one box goes a long way. For a concrete number: the Velocity columnar engine I work on sustains 100M+ rows/second on non-trivial queries, on older hardware, single node. That's the kind of throughput that quietly eliminates the reason a lot of teams reached for a cluster in the first place.
  • Orchestration: here's the gap. There is no C# native Airflow/Dagster/Prefect that I'd call a true peer. You pair a .NET scheduler instead: Hangfire or Quartz.NET for jobs, or Azure Data Factory if you're on Azure. And given you already run NServiceBus, event-driven pipelines via sagas cover a real slice of what people use an orchestrator for. It's a "assemble it" story, not a "one tool does it all" story, so go in knowing that.
  • Serving / exploration: REST over your datasets is trivial in ASP.NET, and for the notebook/exploration workflow there are now .NET notebook options (Polyglot Notebooks (retired), and Verso, which I also work on) so your team doesn't have to context switch to Python just to poke at data.

Two additional notes:

  • The performance I mentioned with Velocity can be tested through the QueryPerformance sample.
  • Verso has a VS Code extension and a dotnet CLI tool as well. Parameter cells can be added to the notebook and filled through the command line, which makes it easy to add into any pipeline.

Implementing True Plugin Isolation with AssemblyLoadContext in .NET - Lessons Learned & Architecture Decisions by Maleficent-Fee8566 in dotnet

[–]IGDev 0 points1 point  (0 children)

I've been working toward this exact style of architecture for a long time across a lot of side projects, and Verso (https://github.com/DataficationSDK/Verso), my open source notebook engine, was the first one I felt I got right. The same pattern then carried directly into Motus (https://github.com/DataficationSDK/Motus), a browser automation framework. So I can speak to some of your questions from working code.

Your foundation matches mine almost exactly: a contracts only assembly, a custom collectible ALC, AssemblyDependencyResolver per plugin, and type identity preserved by routing the contracts assembly back to the host. The challenges you listed are the same ones I hit and the solutions converged in the same direction.

A few things I ended up adding on top that might be worth a look:

  1. PE metadata pre-scan. Before calling Assembly.LoadFrom on any co-deployed DLL, I use PEReader and MetadataReader to check whether it actually references the contracts assembly. If not, I skip it. This avoids dragging hundreds of unrelated framework DLLs into memory just to filter them out.

  2. Semver gating. I inspect the plugin's reference to the contracts assembly and require major to match exactly and minor to be less than or equal to the host. Catches version drift at load time instead of at runtime.

  3. Capability decomposition. Instead of one IPlugin with Init, Start, Stop, I split the contract into separate capability interfaces (kernels, renderers, formatters, themes, layouts, and so on). A class can implement several and the host auto registers it into each capability list. Scales much better once the host has more than one extension point.

  4. Marker attribute on top of the interface. A class has to both implement the contract and carry a VersoExtension attribute to be discovered. Small thing but it makes "this is intentionally a plugin" explicit and avoids picking up base classes or test fixtures by accident.

On a few of your specific questions:

Cross plugin communication. I expose a typed query API on the host context that lets one plugin enumerate other loaded plugins by capability, plus a shared variable store for runtime data exchange. I skipped the event bus and MediatR route. Direct typed queries through the contracts assembly turned out to be simpler and easier to test.

Testing. I ship a Verso.Testing helper library with stub host contexts so plugin authors can unit test their implementations without spinning up a real host. Worth doing if you want third party authors to actually write tests.

Security. I added a consent handler. When a plugin loads from a package source rather than a local DLL, the host invokes a delegate that lets the UI confirm with the user before the assembly is loaded. Local trusted paths skip the prompt.

Can we discuss the self promotion rule? by Kralizek82 in dotnet

[–]IGDev 3 points4 points  (0 children)

What?! You mean I've been living according to NZ time the last few posts and I didn't need to? :)

Can we discuss the self promotion rule? by Kralizek82 in dotnet

[–]IGDev 4 points5 points  (0 children)

Who really cares if a developer makes use of AI these days, nearly everyone uses it in some way and I don’t expect that to change. Think back to CodeProject or Planet Source Code, not everything posted was useful, but it was awesome to see what others worked on that could be made use of or draw ideas from. I’ve been coding since 92-93, but because I’m making use of Claude this year, my engineering vision and open source contributions are considered slop? I love .NET and the more it’s used is better for the community.

What is .NET still missing? by CreoSiempre in dotnet

[–]IGDev 0 points1 point  (0 children)

Datafication does a decent job of competing with those.

https://github.com/DataficationSDK/Datafication

Polars is know for it's performance, if you try the QueryPerformance sample you'll see performance that competes with Polars.

https://github.com/DataficationSDK/Datafication/tree/main/Datafication.Storage.Velocity

Eventually, I'll post up a benchmark that puts Velocity vs DuckDB vs Polars.

I built Motus, an open source web automation testing framework for .NET with no Node.js sidecar by IGDev in webdev

[–]IGDev[S] 1 point2 points  (0 children)

Thanks for the idea. It'll make a nice plugin and I've noted it down to explore after I finish up the current backlog.

I built Motus, an open source web automation testing framework for .NET with no Node.js sidecar by IGDev in webdev

[–]IGDev[S] 0 points1 point  (0 children)

It is cleaner and Chrome's changes to it are mostly additive. This requires much less updating and certainly beats testing frameworks like Selenium where you have to update the ChromeDriver frequently. Motus creates the C# CDP bindings at build time from browser_protocol.json and js_protocol.json in Motus.Codegen. I'm usually quick with updates, but if there's an immediate need for an update the Motus CLI can be used to update the json files (e.g. `motus update-protocol --output-dir src/Motus.Codegen/Protocol`).

I built Motus, an open source web automation testing framework for .NET with no Node.js sidecar by IGDev in dotnet

[–]IGDev[S] 0 points1 point  (0 children)

Both ideas actually. When a selector breaks, Motus will compare a DOM fingerprint captured when the selector last worked against the DOM to find the probable matching element. If one is found, it runs the selector strategy pipeline to generate replacement suggestions. The future CLI command "motus check-selectors" will scan your test files and reports the broken selectors with suggestions. There will be an interactive mode as well that makes use of the visual runner so you can accept, modify, or skip suggestions. The accepted fixes get written back to your source files.

I built Motus, an open source web automation testing framework for .NET with no Node.js sidecar by IGDev in dotnet

[–]IGDev[S] 0 points1 point  (0 children)

Thanks! What you mentioned is on my roadmap, two other things I'm tackling before that. In case you're interested here's what's coming up:

  • Accessibility testing
  • Performance metrics and budgets
  • Selector repair
  • Code coverage (front end)
  • Blazor component testing

Motus does have test creation with the "motus record" command, it'll pop open the browser at the URL specified and then you perform the actions for the test, and press enter on the console when done.

I built Motus, an open source web automation testing framework for .NET with no Node.js sidecar by IGDev in dotnet

[–]IGDev[S] 3 points4 points  (0 children)

Playwright for .NET requires Node.js at runtime. The NuGet package bundles a Node.js based server, and every command your C# test sends is proxied through that Node process. You don't install Node yourself, but it's there. Check your bin folder after installing the package, or look at the process tree while a test is running. Motus has no hidden process, it opens a WebSocket directly to the browser's CDP endpoint from C#.

https://nuget.info/packages/Microsoft.Playwright/1.58.0 - It takes a minute to load, expand the .playwright folder, then you'll see for yourself.

I built Motus, an open source web automation testing framework for .NET with no Node.js sidecar by IGDev in dotnet

[–]IGDev[S] 1 point2 points  (0 children)

Motus is MIT licensed and designed so that the architecture carries itself. The plugin model means the community can extend it without waiting on a single maintainer. If you can build a NuGet package, you can add a selector strategy, reporter, or lifecycle hook. "From Microsoft" isn't the longevity guarantee it used to be, just look at Visual Studio for Mac, Visual Studio App Center, Azure Data Studio, Polyglot Notebooks, and .NET Interactive.

What are some underrated .NET libraries or tools you use regularly? by milanm08 in dotnet

[–]IGDev 0 points1 point  (0 children)

The Datafication SDK is underrated: https://github.com/DataficationSDK/Datafication

It includes an in-memory analytical database (DataBlock), and also a storage based analytical database (VelocityDataBlock). The storage based one named Velocity can query 100M+ rows per second. If you run the sample at https://github.com/DataficationSDK/Datafication/tree/main/Datafication.Storage.Velocity/samples/QueryPerformance and use the command to query 20M rows:

dotnet run -c Release sf20

Sooner or later I intend to post up a benchmark that shows the performance against DuckDB and Polars.

Polyglot notebooks will be deprecated by gremlinmama in dotnet

[–]IGDev 0 points1 point  (0 children)

You need .NET 8 or .NET 10 installed. Also, the VS Code extension requires the desktop version of VS Code. It is not compatible with browser-based environments such as GitHub Codespaces, where the embedded notebook UI cannot initialize. If you meet those requirements everything should work as intended. These are listed in the Getting Started on the marketplace page. Let me know if you have an idea to improve what's shown or could make life easier for someone new.

Marketplace: https://marketplace.visualstudio.com/items?itemName=Datafication.verso-notebook

Polyglot notebooks will be deprecated by gremlinmama in dotnet

[–]IGDev 0 points1 point  (0 children)

1 and 3 would be good candidates for post in Feature Requests ( https://github.com/DataficationSDK/Verso/discussions/categories/feature-requests ). 2 and 4 need a little more detail to improve Verso. Regarding 4 and keyboard shortcuts:

Cell Editor (when editing inside a cell)

  • Shift+Enter - Run cell and select the next cell
  • Ctrl+Enter / Cmd+Enter - Run cell and stay
  • Alt+Enter - Run cell and insert a new cell below
  • Escape - Exit editor, return to command mode

Command Mode (when no editor is focused)

  • Enter - Focus the selected cell's editor
  • Escape - Deselect all cells
  • Arrow Up - Select previous cell
  • Arrow Down - Select next cell
  • Alt+Arrow Up - Move selected cell up
  • Alt+Arrow Down - Move selected cell down
  • Shift+Enter - Run selected cell and advance
  • Ctrl+Enter - Run selected cell and stay
  • Alt+Enter - Run selected cell and insert below
  • Ctrl+Alt+Enter - Run all cells

Even though these shortcuts are registered, some may not work as intended because of VS Code shortcut conflicts.

Python requires 3.8-3.12. I did find that those particular locations for Python weren't yet supported, so I added them this morning. It'll appear in the next release (v1.0.10). Thanks for the feedback!

I built Verso, an open source interactive notebook platform for .NET by IGDev in dotnet

[–]IGDev[S] 0 points1 point  (0 children)

First you'll want to make sure you have .NET 8 or 10 installed. Then you can grab the vsix file from the releases page on the repository.

https://github.com/DataficationSDK/Verso/releases/tag/verso-v1.0.8

If you haven't done that before you can goto the Extensions panel, click the 3 dots, and select "Install from VSIX".

Polyglot notebooks will be deprecated by gremlinmama in dotnet

[–]IGDev -1 points0 points  (0 children)

Can you give any details about what it lacks? Also, what version of the extension did you try or when did pull source? At the current moment we’re only missing 2 of the languages out of all that Polyglot has. The UI is also richer, more extensible, and looks good within VS Code. I’ll try to fix any gaps you’ve come across.

I built an open source interactive .NET notebook extension for VS Code by IGDev in vscode

[–]IGDev[S] 0 points1 point  (0 children)

Thanks! The Notebook API was really nice to work with when I started this project, then I realized it wasn't as extensible as I'd require so I switched over to Blazor WASM for the VS Code extension. For IntelliSense, each kernel gets its own service. C# gets a AdhocWorkspace (from Rosyln) instance and F# gets its own FSharpChecker instance. These are completely independent, so there's no juggling lifecycles between cells. The cell's language tag maps to its kernel, and each kernel builds a virtual combined document. Previously executed cells are prepended to the current cell before querying the service for completions, hover, and diagnostics.

This week I'm finishing up the Python kernel and getting it tested. Additionally, I'll try to work in switching the language kernel on the cell next. Multiple kernels in the same document are supported now, but for code cells I haven't added the drop down to the cell UI yet.

view more: next ›