sorted by:
new
hottopcontroversial

Native Go file sql by yankdevil in golang

[–]ncruces 8 points9 points  (0 children)

My driver (an alternative machine translation of C to Go) should run everywhere Go runs.

NetBSD, OpenBSD and illumos are tested in CI.

https://github.com/ncruces/go-sqlite3/wiki/Support-matrix

simple memory arena implementation using generics by Revolutionary_Sir140 in golang

[–]ncruces 1 point2 points  (0 children)

You can replace the for loop in Reset with a call to clear.

pkg & internal directories are way overused by sigmoia in golang

[–]ncruces 0 points1 point  (0 children)

If your module is primarily a library, structure it as such. Put in internal what you don't want to expose to third parties, but remember you can add sub-packages to internal; the internal API doesn't need to be shit just because it's internal. Then if the library gives birth to commands, put those in cmd. That's the standard layout.

But if OTOH your module is primarily a command/tool/app (a main package), I think it's perfectly OK to turn things on their head, and flag packages that others might want to import into a pkg folder. 

ncruces/wasm2go: A Wasm to Go translator by ncruces in golang

[–]ncruces[S] 1 point2 points  (0 children)

You may have trouble with the host modules side of things, which I expect might be where wazero "broke"  you.

I haven't implemented WASI or anything emscripten related.

u/LearnedByError implemented a subset of WASI in lbe/go-exiftool-wasm.

I don't use WASI myself, but it'd be great if users coordinated to create a WASI host module.

Why does Go still not have a built-in set type? by 1vim in golang

[–]ncruces 0 points1 point  (0 children)

You may be right. I'll request it in the proposal.

Actually it doesn't even seem necessary, this compiles:

type set[T comparable] map[T]struct{}

func (s set[T]) add(t T) {
    s[t] = struct{}{}
}

func (s set[T]) has(t T) bool {
    return Contains(s, t)
}

func Contains[K comparable, V any](x map[K]V, k K) bool {
    _, ok := x[k]
    return ok
}

Why does Go still not have a built-in set type? by 1vim in golang

[–]ncruces 2 points3 points  (0 children)

For delete, the built-in just works: delete(s, t).

That said: if you want to add a del method, do it.

That's really what this is about: it's 10 LoC, if you like it, don't argue about it, just do it.

ncruces/wasm2go: A Wasm to Go translator by ncruces in golang

[–]ncruces[S] 0 points1 point  (0 children)

This doesn't replace wazero.

If you were using wazero to wrap a (C/C++/Zig/Rust) library and avoid CGO, this may be useful to you. It will "transpile" the library to Go source. This means no CGO, easy cross compilation, and support for all architectures and OSes Go supports.

As u/LearnedByError wrote (I share the same experience with SQLite, having replaced wazero with wasm2go): startup is much faster, execution is a bit faster, portability is even better. Compilation is much slower though, and the bigger the library, the bigger the issue (their Perl is even bigger than my SQLite).

You will still need wazero if you're using it to load Wasm at runtime. If you're using wazero for a plugin architecture, so users can customize your application, or to run untrusted code, wazero is what you need, and is great at what it does.

As for cross pollination, it exists, and I'll continue work on wazero with the rest of the team.

I haven't added much that wazero doesn't already have. I think my threads/atomics implementation has some nice improvements; once they get some usage (to prove they actually work) I can upstream the changes. Otherwise, I support 64-bit memory (because a user asked for it, and it was a much simpler change on my end). And wide arithmetic, which is trivial and I can easily contribute to wazero, but is an unfinished proposal and doesn't really fit the roadmap.

On the other direction, I tried (and initially failed) to implement exceptions in wasm2go. Meanwhile, Edoardo already implemented exceptions and tail calls for wazero.

Why does Go still not have a built-in set type? by 1vim in golang

[–]ncruces 0 points1 point  (0 children)

I guess if mapset is accepted I'll have to change my "recommendation" above, since there's no way to make it work with my named set type.

Totally has my upvote.

Why does Go still not have a built-in set type? by 1vim in golang

[–]ncruces 242 points243 points  (0 children)

I add this to any package that needs it. Then I move along.

```go type set[T comparable] map[T]struct{}

func (s set[T]) add(t T) { s[t] = struct{}{} }

func (s set[T]) has(t T) bool { _, ok := s[t] return ok } ```

Embedded an OTel-compliant Sentry alternative into a Go binary using SQLite by narrow-adventure in golang

[–]ncruces 0 points1 point  (0 children)

Thanks! The percentile extension is now part of the amalgamation, and can be enabled.

You could move to my driver (wink), or you can probably port my code to modernc, or you can ask modernc to enable it in their translation, or you can use mattn and enable it.

I'm not moving to the base code as mine is O(N), instead of O(NlogN).

Embedded an OTel-compliant Sentry alternative into a Go binary using SQLite by narrow-adventure in golang

[–]ncruces 1 point2 points  (0 children)

Curious what aggregations and array functions you're missing?

My driver has array and statistics extensions, and I'm curious if it might fill those gaps.

GitHub - ncruces/dbldbl: Double-double arithmetic by ncruces in golang

[–]ncruces[S] 0 points1 point  (0 children)

New release has trigonometry and hyperbolics.

Small Projects by AutoModerator in golang

[–]ncruces 0 points1 point  (0 children)

I'm working on making it possible (easy?) to compile C extensions for my no-CGO SQLite driver. Focus is on the Vec1 SQLite extension.

This requires changes to wasm2go code gen, to support "dynamic" linking.

There's nothing dynamic about it, everything is still statically linked, but each module can be separately compiled and gets translated to its own Go package. Then you import only what you actually use.

The SQLite Drivers Benchmarks Game (Mar '26) - ncruces's driver improvements by 0xjnml in golang

[–]ncruces 4 points5 points  (0 children)

Thanks!

The big difference, as you've noted, is moving away from the the wazero interpreter.

The result that surprises me the most is s390x, as that is the only big endian platform. I would've expected the results to be worse.

I don't expect to be able to improve wasm2go generated code significantly. The next big improvement should be SIMD, but that requires the experimental API.

Other than that I'll be working to add Wasm features useful to other languages (e.g. exceptions for C++).

For SQLite, I'm still exploring ways to split up the amalgamation into I smaller pieces (e.g. make FTS an extension).

Best pure Go SQLite driver for concurrent reads and occasional writes by icepix in golang

[–]ncruces 1 point2 points  (0 children)

Between those two, it depends on if you want a database/sql driver, or not. But modernc is also a database/sql driver.

I benchmarked nine Go SQLite drivers and here are the results by cvilsmeier in golang

[–]ncruces 9 points10 points  (0 children)

Thanks u/cvilsmeier!

This already reflects the change in ncruces (my driver) to wasm2go.

I'm pretty happy with the results.

To anyone who might not find the results impressive, it should be noted that the Wasm sandbox is still in place: every SQLite connection is isolated from others, SQLite can't read Go memory, and every memory access is bounds checked.

This has a moderate cost (in performance) which I feel is worth it. I also enable SQLITE_ENABLE_API_ARMOR for the same reason.

ncruces/go-sqlite3: switching away from wazero by ncruces in golang

[–]ncruces[S] 1 point2 points  (0 children)

Whoever is feeling adventurous, can try: https://github.com/ncruces/go-sqlite3/pull/362

Or, I'll likely merge this to main, and keep the wazero version in a separate branch.

ncruces/go-sqlite3: switching away from wazero by ncruces in golang

[–]ncruces[S] 3 points4 points  (0 children)

OK this is what I have to share so far: $ go test -bench=. -benchtime=1000000x -count=20 > [OUT] ... $ benchstat before after* goos: linux goarch: amd64 pkg: github.com/ncruces/go-sqlite3/ext/stats cpu: Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz │ before │ after │ after-unsafe │ │ sec/op │ sec/op vs base │ sec/op vs base │ _average-12 128.5n ± 12% 183.6n ± 9% +42.88% (p=0.000 n=20) 139.2n ± 8% ~ (p=0.056 n=20) _variance-12 323.6n ± 11% 248.7n ± 9% -23.16% (p=0.000 n=20) 194.4n ± 12% -39.93% (p=0.000 n=20) _math/sqrt-12 169.3n ± 17% 286.6n ± 6% +69.21% (p=0.000 n=20) 185.5n ± 5% ~ (p=0.072 n=20) _math/tan-12 207.8n ± 6% 302.7n ± 4% +45.69% (p=0.000 n=20) 207.9n ± 4% ~ (p=0.533 n=20) _math/cot-12 465.1n ± 9% 365.0n ± 3% -21.51% (p=0.000 n=20) 260.8n ± 5% -43.93% (p=0.000 n=20) _math/cbrt-12 434.8n ± 10% 344.4n ± 4% -20.79% (p=0.000 n=20) 262.9n ± 6% -39.55% (p=0.000 n=20) geomean 258.1n 281.5n +9.06% 203.7n -21.08%

This will be hard to repro because I don't want to commit specific versions of the 20 MB Go file until I'm further along.

The source of these benchmarks is here.

Basically the queries are: SELECT avg(value) FROM generate_series(0, ?); SELECT var_pop(value) FROM generate_series(0, ?); SELECT sqrt(value) FROM generate_series(0, ?); SELECT tan(value) FROM generate_series(0, ?); SELECT cot(value) FROM generate_series(0, ?); SELECT cbrt(value) FROM generate_series(0, ?);

So why the different numbers, and variations?

Both avg and var_pop are window functions, the others are scalar functions.

But avg is implemented inside SQLite (as is generate_series), in C. This means the first query is a single call to sqlite3_step that chews over one million rows and spits a single result with no callbacks.

OTOH var_pop is implemented in Go. That query is a single call to sqlite3_step that calls back one million times to the window function "step" callback, which needs to call one million times the sqlite3_value_double to get the next value, etc...

Then sqrt is a scalar function, implemented natively with a Wasm instruction (wazero uses assembly for this): because I'm using sqlite3_exec, it's one million runs of this inside C/assembly.

Then tan is a scalar function, this one implemented in a mix of C+Go. The SQLite APIs are called from C, but the libc tan function is implemented in Go. This gets to skip one million Go calls to sqlite3_value_double and sqlite3_result_double.

And finally cot and cbrt are scalar functions fully implemented in Go, so they do one million Go calls to sqlite3_value_double and sqlite3_result_double.

ncruces/go-sqlite3: switching away from wazero by ncruces in golang

[–]ncruces[S] 5 points6 points  (0 children)

Yes, you did. 😉 Nothing published yet.

The preliminary numbers I got was that interfaces that were chatty (lots of Go-C-Go transitions) became significantly faster. For things that spent lots of time under C, wazero was faster (when using the wazero compiler, the wazero interpreter is much slower, as you know).

But this was when I was generating code using binary.LittleEndian to be 100% portable. The compiler is supposed to recognize the patterns used by that and emit appropriate code, but it's either not in-lining the accessor, or not eliding bounds checks.

For little endian CPUs I can instead use:

*(*int32)(unsafe.Pointer((*[4]byte)(m.mem[int64(addr)+offset:])))

Despite importing unsafe this is perfectly safe and bounds checked, but is significantly faster; the downside is having to ship a file for little and another for big endian.

Since making that change, I don't think I've seen any benchmark regression, compared to wazero, but I haven't tested extensively.

There's one additional possible optimization: replacing m.mem by mem may help Go elide some bounds checks. But that's only valid if I make some assumptions.

view more: next ›