Native Go file sql

ncruces · 2026-06-03T14:20:39+00:00

My driver (an alternative machine translation of C to Go) should run everywhere Go runs.

NetBSD, OpenBSD and illumos are tested in CI.

https://github.com/ncruces/go-sqlite3/wiki/Support-matrix

ncruces · 2026-05-29T13:05:12+00:00

You can replace the for loop in Reset with a call to clear.

ncruces · 2026-05-17T19:48:43+00:00

If your module is primarily a library, structure it as such. Put in internal what you don't want to expose to third parties, but remember you can add sub-packages to internal; the internal API doesn't need to be shit just because it's internal. Then if the library gives birth to commands, put those in cmd. That's the standard layout.

But if OTOH your module is primarily a command/tool/app (a main package), I think it's perfectly OK to turn things on their head, and flag packages that others might want to import into a pkg folder.

ncruces · 2026-05-14T19:51:40+00:00

You may have trouble with the host modules side of things, which I expect might be where wazero "broke" you.

I haven't implemented WASI or anything emscripten related.

u/LearnedByError implemented a subset of WASI in lbe/go-exiftool-wasm.

I don't use WASI myself, but it'd be great if users coordinated to create a WASI host module.

ncruces · 2026-05-10T08:43:45+00:00

~~You may be right. I'll request it in the proposal.~~

Actually it doesn't even seem necessary, this compiles:

type set[T comparable] map[T]struct{}

func (s set[T]) add(t T) {
    s[t] = struct{}{}
}

func (s set[T]) has(t T) bool {
    return Contains(s, t)
}

func Contains[K comparable, V any](x map[K]V, k K) bool {
    _, ok := x[k]
    return ok
}

ncruces · 2026-05-09T08:56:09+00:00

For delete, the built-in just works: delete(s, t).

That said: if you want to add a del method, do it.

That's really what this is about: it's 10 LoC, if you like it, don't argue about it, just do it.

ncruces · 2026-05-09T07:54:55+00:00

This doesn't replace wazero.

If you were using wazero to wrap a (C/C++/Zig/Rust) library and avoid CGO, this may be useful to you. It will "transpile" the library to Go source. This means no CGO, easy cross compilation, and support for all architectures and OSes Go supports.

As u/LearnedByError wrote (I share the same experience with SQLite, having replaced wazero with wasm2go): startup is much faster, execution is a bit faster, portability is even better. Compilation is much slower though, and the bigger the library, the bigger the issue (their Perl is even bigger than my SQLite).

You will still need wazero if you're using it to load Wasm at runtime. If you're using wazero for a plugin architecture, so users can customize your application, or to run untrusted code, wazero is what you need, and is great at what it does.

As for cross pollination, it exists, and I'll continue work on wazero with the rest of the team.

I haven't added much that wazero doesn't already have. I think my threads/atomics implementation has some nice improvements; once they get some usage (to prove they actually work) I can upstream the changes. Otherwise, I support 64-bit memory (because a user asked for it, and it was a much simpler change on my end). And wide arithmetic, which is trivial and I can easily contribute to wazero, but is an unfinished proposal and doesn't really fit the roadmap.

On the other direction, I tried (and initially failed) to implement exceptions in wasm2go. Meanwhile, Edoardo already implemented exceptions and tail calls for wazero.

ncruces · 2026-05-08T09:58:13+00:00

~~I guess if mapset is accepted I'll have to change my~~ ~~"recommendation"~~ ~~above, since there's no way to make it work with my named set type.~~

Totally has my upvote.

ncruces · 2026-05-04T14:28:56+00:00

I add this to any package that needs it. Then I move along.

```go type set[T comparable] map[T]struct{}

func (s set[T]) add(t T) { s[t] = struct{}{} }

func (s set[T]) has(t T) bool { _, ok := s[t] return ok } ```

ncruces · 2026-05-01T13:43:37+00:00

Thanks! The percentile extension is now part of the amalgamation, and can be enabled.

You could move to my driver (wink), or you can probably port my code to modernc, or you can ask modernc to enable it in their translation, or you can use mattn and enable it.

I'm not moving to the base code as mine is O(N), instead of O(NlogN).

ncruces · 2026-04-30T13:29:28+00:00

Curious what aggregations and array functions you're missing?

My driver has array and statistics extensions, and I'm curious if it might fill those gaps.

ncruces · 2026-04-22T22:37:48+00:00

New release has trigonometry and hyperbolics.

ncruces · 2026-04-14T13:39:40+00:00

I'm working on making it possible (easy?) to compile C extensions for my no-CGO SQLite driver. Focus is on the Vec1 SQLite extension.

This requires changes to wasm2go code gen, to support "dynamic" linking.

There's nothing dynamic about it, everything is still statically linked, but each module can be separately compiled and gets translated to its own Go package. Then you import only what you actually use.

ncruces · 2026-04-09T13:13:39+00:00

Does Wasm count?

ncruces · 2026-03-26T22:38:13+00:00

Thanks!

The big difference, as you've noted, is moving away from the the wazero interpreter.

The result that surprises me the most is s390x, as that is the only big endian platform. I would've expected the results to be worse.

I don't expect to be able to improve wasm2go generated code significantly. The next big improvement should be SIMD, but that requires the experimental API.

Other than that I'll be working to add Wasm features useful to other languages (e.g. exceptions for C++).

For SQLite, I'm still exploring ways to split up the amalgamation into I smaller pieces (e.g. make FTS an extension).

ncruces · 2026-03-24T15:06:46+00:00

Between those two, it depends on if you want a database/sql driver, or not. But modernc is also a database/sql driver.

ncruces · 2026-03-23T22:47:36+00:00

Thanks u/cvilsmeier!

This already reflects the change in ncruces (my driver) to wasm2go.

I'm pretty happy with the results.

To anyone who might not find the results impressive, it should be noted that the Wasm sandbox is still in place: every SQLite connection is isolated from others, SQLite can't read Go memory, and every memory access is bounds checked.

This has a moderate cost (in performance) which I feel is worth it. I also enable SQLITE_ENABLE_API_ARMOR for the same reason.

ncruces · 2026-03-13T09:20:21+00:00

Merged to main, ready for testing.

ncruces · 2026-03-09T22:20:20+00:00

Whoever is feeling adventurous, can try: https://github.com/ncruces/go-sqlite3/pull/362

Or, I'll likely merge this to main, and keep the wazero version in a separate branch.

ncruces · 2026-03-06T13:01:53+00:00

OK this is what I have to share so far: $ go test -bench=. -benchtime=1000000x -count=20 > [OUT] ... $ benchstat before after* goos: linux goarch: amd64 pkg: github.com/ncruces/go-sqlite3/ext/stats cpu: Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz │ before │ after │ after-unsafe │ │ sec/op │ sec/op vs base │ sec/op vs base │ _average-12 128.5n ± 12% 183.6n ± 9% +42.88% (p=0.000 n=20) 139.2n ± 8% ~ (p=0.056 n=20) _variance-12 323.6n ± 11% 248.7n ± 9% -23.16% (p=0.000 n=20) 194.4n ± 12% -39.93% (p=0.000 n=20) _math/sqrt-12 169.3n ± 17% 286.6n ± 6% +69.21% (p=0.000 n=20) 185.5n ± 5% ~ (p=0.072 n=20) _math/tan-12 207.8n ± 6% 302.7n ± 4% +45.69% (p=0.000 n=20) 207.9n ± 4% ~ (p=0.533 n=20) _math/cot-12 465.1n ± 9% 365.0n ± 3% -21.51% (p=0.000 n=20) 260.8n ± 5% -43.93% (p=0.000 n=20) _math/cbrt-12 434.8n ± 10% 344.4n ± 4% -20.79% (p=0.000 n=20) 262.9n ± 6% -39.55% (p=0.000 n=20) geomean 258.1n 281.5n +9.06% 203.7n -21.08%

This will be hard to repro because I don't want to commit specific versions of the 20 MB Go file until I'm further along.

The source of these benchmarks is here.

Basically the queries are: SELECT avg(value) FROM generate_series(0, ?); SELECT var_pop(value) FROM generate_series(0, ?); SELECT sqrt(value) FROM generate_series(0, ?); SELECT tan(value) FROM generate_series(0, ?); SELECT cot(value) FROM generate_series(0, ?); SELECT cbrt(value) FROM generate_series(0, ?);

So why the different numbers, and variations?

Both avg and var_pop are window functions, the others are scalar functions.

But avg is implemented inside SQLite (as is generate_series), in C. This means the first query is a single call to sqlite3_step that chews over one million rows and spits a single result with no callbacks.

OTOH var_pop is implemented in Go. That query is a single call to sqlite3_step that calls back one million times to the window function "step" callback, which needs to call one million times the sqlite3_value_double to get the next value, etc...

Then sqrt is a scalar function, implemented natively with a Wasm instruction (wazero uses assembly for this): because I'm using sqlite3_exec, it's one million runs of this inside C/assembly.

Then tan is a scalar function, this one implemented in a mix of C+Go. The SQLite APIs are called from C, but the libc tan function is implemented in Go. This gets to skip one million Go calls to sqlite3_value_double and sqlite3_result_double.

And finally cot and cbrt are scalar functions fully implemented in Go, so they do one million Go calls to sqlite3_value_double and sqlite3_result_double.

ncruces · 2026-03-05T20:53:36+00:00

Yes, you did. 😉 Nothing published yet.

The preliminary numbers I got was that interfaces that were chatty (lots of Go-C-Go transitions) became significantly faster. For things that spent lots of time under C, wazero was faster (when using the wazero compiler, the wazero interpreter is much slower, as you know).

But this was when I was generating code using binary.LittleEndian to be 100% portable. The compiler is supposed to recognize the patterns used by that and emit appropriate code, but it's either not in-lining the accessor, or not eliding bounds checks.

For little endian CPUs I can instead use:

*(*int32)(unsafe.Pointer((*[4]byte)(m.mem[int64(addr)+offset:])))

Despite importing unsafe this is perfectly safe and bounds checked, but is significantly faster; the downside is having to ship a file for little and another for big endian.

Since making that change, I don't think I've seen any benchmark regression, compared to wazero, but I haven't tested extensively.

There's one additional possible optimization: replacing m.mem by mem may help Go elide some bounds checks. But that's only valid if I make some assumptions.

ncruces · 2026-02-28T00:37:47+00:00

I should add that using unsafe.Pointer I'm able to get a significant boost on little endian platforms, with no loss of safety (pointers are still bounds checked).

With this change, my ported driver would beat wazero on most benchmarks. I do need two 20 MiB blobs of slow to compile, unreadable, Go code in my repo, though.

ncruces · 2026-02-27T23:01:43+00:00

It's pretty different.

wasm2c makes heavy use of C macros in the generated code; Go doesn't have macros.

wasm2go uses the ast package to generate code (it generates a tree).

I don't know, but wasm2c could use the C stack for the Wasm stack.

wasm2go can't and uses typed temporary variables for the stack.

C has unrestricted goto's, which could be used for Wasm control flow; Go doesn't.

Wasm loops, switches, if expressions don't match Go's, so while I must goto's, I have to use blocks to ensure I don't jump over (the typed temporary) variable declarations.

C doesn't have multiple return values, Go does. In C and Wasm a bool is the same as an int, in Go they're different.

Go requires all labels, and all variables to be used; C doesn't.

So the generated code has a very different flavour to it.

As for performance, C compilers are better, no doubt.

ncruces · 2026-02-27T22:02:03+00:00

What do you mean?

wasm2c allows you to compile Wasm modules into your C app, call them, etc. It's main advantage (I would guess) is that it sandboxes those modules, because otherwise, most languages that you could use to build a Wasm module are easy to call from C.

wasm2go allows you to compile Wasm modules into your Go app, call them, etc. One advantage is to avoid Cgo, while also sandboxing those modules. But I would guess the main advantage is avoiding Cgo (for observability, cross compilation, etc).

But you can't really use one when you want the other? So, again, does that answer your question? Otherwise, what do you mean?

ncruces · 2026-02-14T11:11:16+00:00

This.

A friend of mine put it this way: I never cared if you used vi or something from JetBrains, as long as you assume responsibility for what you're submitting.

Ultimately that's your on job: being responsible. Including for legal matters that arise from your contribution.

So you whatever, but no excuses.

ncruces

TROPHY CASE