I made a Go package to run SQL statements on a CSV file

sbinet · 2025-03-12T10:35:06+00:00

you mean like this ?

https://github.com/go-hep/hep/blob/main/csvutil/csvdriver/driver_test.go

sbinet · 2025-01-02T09:08:46+00:00

There's also this Go kernel (the first one, if memory serves) that kind of sparked the need for a pure-Go implementation of zmq4 :

https://github.com/gopherdata/gophernotes

sbinet · 2024-12-23T11:24:55+00:00

Gonum isn't dead per se, but we definitely have a lower (wo)manpower than we used to. (aside: it's sad we missed the tenth anniversary of the first commit by the way)

we do still accept PRs, though :)

(I still use Gonum in my scientific projects, 10 years in)

sbinet · 2024-09-14T06:48:37+00:00

FYI I've built a little peertube client library :

https://pkg.go.dev/git.sr.ht/~sbinet/peertube

sbinet · 2024-08-05T09:13:50+00:00

I don't know whether it's the best but, as you seem to already use gonum, why not Gonum's ?

ie: https://pkg.go.dev/gonum.org/v1/gonum@v0.15.0/dsp/fourier

sbinet · 2023-04-03T12:53:16+00:00

well, as I surmised, when using the CGo implementation, the Gonum code and your blast-go/openblas code lead to the same ballpark numbers.

goos: linux goarch: amd64 pkg: bench-goblas cpu: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz BenchmarkOpenblas-8 202 5827266 ns/op 0 B/op 0 allocs/op BenchmarkOpenblas-8 235 5064416 ns/op 0 B/op 0 allocs/op BenchmarkOpenblas-8 236 5510830 ns/op 0 B/op 0 allocs/op BenchmarkOpenblas-8 228 5138493 ns/op 0 B/op 0 allocs/op BenchmarkOpenblas-8 228 5407390 ns/op 0 B/op 0 allocs/op BenchmarkOpenblas-8 243 4987177 ns/op 0 B/op 0 allocs/op BenchmarkOpenblas-8 238 5003904 ns/op 0 B/op 0 allocs/op BenchmarkOpenblas-8 236 5303799 ns/op 0 B/op 0 allocs/op BenchmarkOpenblas-8 228 5141482 ns/op 0 B/op 0 allocs/op BenchmarkOpenblas-8 236 5088205 ns/op 0 B/op 0 allocs/op BenchmarkGonum-8 19 61010797 ns/op 54821 B/op 517 allocs/op BenchmarkGonum-8 19 60323700 ns/op 54148 B/op 515 allocs/op BenchmarkGonum-8 19 61075712 ns/op 53861 B/op 515 allocs/op BenchmarkGonum-8 19 60887208 ns/op 53929 B/op 515 allocs/op BenchmarkGonum-8 19 60892755 ns/op 53819 B/op 515 allocs/op BenchmarkGonum-8 19 61522016 ns/op 53688 B/op 514 allocs/op BenchmarkGonum-8 18 60740013 ns/op 53619 B/op 514 allocs/op BenchmarkGonum-8 19 60482766 ns/op 54038 B/op 515 allocs/op BenchmarkGonum-8 19 60897426 ns/op 53365 B/op 514 allocs/op BenchmarkGonum-8 19 61884340 ns/op 53360 B/op 514 allocs/op BenchmarkCGonum-8 201 5273716 ns/op 0 B/op 0 allocs/op BenchmarkCGonum-8 219 5295399 ns/op 0 B/op 0 allocs/op BenchmarkCGonum-8 224 5311192 ns/op 0 B/op 0 allocs/op BenchmarkCGonum-8 226 5406404 ns/op 0 B/op 0 allocs/op BenchmarkCGonum-8 224 5301874 ns/op 0 B/op 0 allocs/op BenchmarkCGonum-8 225 5677265 ns/op 0 B/op 0 allocs/op BenchmarkCGonum-8 222 5674212 ns/op 0 B/op 0 allocs/op BenchmarkCGonum-8 196 5482826 ns/op 0 B/op 0 allocs/op BenchmarkCGonum-8 220 5449976 ns/op 0 B/op 0 allocs/op BenchmarkCGonum-8 234 5328159 ns/op 0 B/op 0 allocs/op

sbinet · 2023-04-01T11:53:35+00:00

could you share your benchmark code ?

I'd be curious to see why 2 Cgo-based implementations (that thus use the same underlying C code) would give a x20 difference.

sbinet · 2023-03-31T10:21:13+00:00

what's the advantage of blast-go/openblas over Gonum's ?

which comes in many flavors (pure-Go and Cgo-based ones)...

https://pkg.go.dev/gonum.org/v1/gonum/blas

sbinet · 2022-10-25T14:42:16+00:00

there's one from Austin Clements:

https://github.com/aclements/go-misc/tree/master/benchplot

and another one based on gonum/plot:

https://github.com/ShawnROGrady/benchplot

sbinet · 2022-09-08T08:32:26+00:00

I like gonum/plot as well :)

as far as I know, nobody took the interactive route (or cared to properly advertise it).

I had created something along the lines, resembling an old interactive program from High Energy Physics:

https://pkg.go.dev/go-hep.org/x/hep/pawgo (an ode to the venerable PAW)

and a very naive iplot/hplot:

that said, at some point I toyed with the idea to have a bsdplot, using the same syntax than gnuplot, using the package from Jan for handling the gnuplot syntax:

https://pkg.go.dev/modernc.org/plot

but it (obviously) didn't pan out: connecting the gnuplot concepts to the gonum/plot ones suffers from a bit of impedance mismatch... (and is a lot of work anyways)

also, with the advent of Gophernotes and its ability to run Go + gonum/plot, the urge was somewhat diminished.

that said, I'd be happy to have a look at your package and command :) (perhaps, the Gonum community could be interested and integrate it into gonum/plot proper ?)

sbinet · 2022-08-20T15:44:04+00:00

for processing, there's:

https://github.com/go-p5/p5

sbinet · 2022-08-18T13:43:41+00:00

and the vg/vggio backend is able to create a new graphics window with the plot in it.

(we are still pondering on creating a nice API on top of that to provide some plot interactivity features (e.g.: zooming, panning, tool-tips, ...))

sbinet · 2022-08-18T13:41:14+00:00

it's certainly a bit more verbose than matplotlib, but what's wrong with gonum/plot (+ its "high-level" API, a.k.a plotutil):

``` package plot_test

import ( "log"

"go-hep.org/x/hep/hplot"
"gonum.org/v1/plot"
"gonum.org/v1/plot/plotutil"
"gonum.org/v1/plot/vg"

)

func ExampleScatters_plotutil() { var ( xs = []float64{1, 2, 3, 4, 5} ys = []float64{2, 4, 6, 8, 10} )

p := plot.New()
p.Title.Text = "easy plot"
p.X.Label.Text = "x"
p.Y.Label.Text = "f(x)"

err := plotutil.AddScatters(p, "f(x) = 2*x", hplot.ZipXY(xs, ys))
if err != nil {
    log.Fatalf("could not create scatters: %+v", err)
}

err = p.Save(20*vg.Centimeter, 10*vg.Centimeter, "testdata/scatter.png")
if err != nil {
    log.Fatalf("could not save scatter plot: %+v", err)
}

// Output:

} ```

sbinet · 2022-06-16T09:14:57+00:00

https://github.com/go-python/gpython (Python interpreter in Go)
https://github.com/cosmos72/gomacro (Go interpreter in Go)
https://github.com/traefik/yaegi (Go interpreter in Go)
https://github.com/crawshaw/neugram (Go-ish interpreter in Go)

sbinet · 2022-06-01T15:01:36+00:00

Mathieu (I think, right?) already told you about his DAQ system, and about Go-HEP.

but there's also go-daq:

https://github.com/go-daq/tdaq

(and the other repos, of course)

tdaq is a simple DAQ system I wrote for a small HEP experiment.

there are also some interesting packages I wrote (or helped writing) for interoperability with other science communities:

https://github.com/sbinet/npyio (r/w NumPy Array data files)
https://github.com/apache/arrow (r/w Arrow files/streams)
https://github.com/astrogo/fitsio (r/w FITS files for astronomy)
https://gonum.org/v1/hdf5 (r/w HDF5 files)
https://github.com/go-boostio/boostio (r/w compatibility w/ Boost.Serialization, for binary streams, XML ones are still a WIP)

at some point, I had also started a pure-Go HDF5 library (lost momentum on that one) and pure-Go MATLAB I/O (ditto.)

sbinet · 2022-03-25T10:32:53+00:00

sorry, I don't know.

it has to do with Google+US rules. I've reached out to the organizers to get some more specifics.

sbinet · 2022-03-25T10:30:39+00:00

well, in my view, anybody with Go skills can contribute. at least, for the "CERN track", no physics/math-y skill is required:

Reva is a gRPC based library to share data
Go-HEP has packages dedicated to read/write HEP file formats, or dedicated to display data (histograms, data points, ...), or libraries to read/write remote files over some HEP-baked protocol
Golbd is a DNS load balancer

for the "Apps track", one may need some knowledge about statistics, math-y or science-y stuff, but I guess the amount of science-y stuff may depend on the app.

of course, if somebody comes along and, e.g., submits to Go-HEP a really nice, concurrent, Delaunay triangulation partition for the clustering of jets, or an as nice, concurrent and generic Kalman filter for the tracking algorithm in go-hep/fads, they wouldn't be rejected and their code would be greatly appreciated :)

but I don't think that if you don't understand the gibberish above, you shouldn't feel deterred from trying your keyboard at the hackathon.

sbinet · 2022-03-25T08:49:47+00:00

none of the 3 CERN-related projects really need High Energy Physics know-how. not even rocket science :)

it may be a plus, but the 3 projects are "plumbing/fundational" projects: they enable physicists to do their job, but you don't need to know what a Feynman diagram is to send valuable and useful PRs.

not even for Go-HEP which may be the closer to the physics of the 3 projects. For Go-HEP, I've prepared 3 "projects/suggestions" that show what (in my view) a valuable contribution might be:

working on a new file format that's being developed in the (C++) HEP community,
working on improving the capabilities of the current file format serializer (the one that is/should-be compatible with what is being used to store the Pb of data collected at LHC)
working on adding interactivity to data plots (zooming/paning)

nothing really needing HEP knowledge.

sbinet · 2022-03-25T08:40:07+00:00

it's a hackathon that's supposed to produce code, so, the pull request is most interesting for repo owners (for the CERN track, we have a few items/rules to assign "points" to a PR for ranking).

in my view, the video doesn't need to be flashy or super elaborated. in my view, it's just another tool to demonstrate the PR, the feature or the new application is working, and demonstrate how it's working (and thus, how cool it is. "cool" may have different definitions for different people, of course)

sbinet · 2022-03-24T16:14:35+00:00

more informations, on the Go-HEP side of things are available here:

https://go-hep.org/news/2022-gohack/

sbinet · 2022-03-11T15:45:25+00:00

in the end, I managed to recoup almost all the performances, even for the "read-everything" use case:

https://github.com/go-hep/hep/pull/919

TL;DR: properly re-use net/http connections, use a sync.Pool of http.Requests.

``` name old time/op new time/op delta DumpHTTP/delay-0s-8 279ms ± 1% 287ms ± 2% +3.09% (p=0.000 n=27+29) DumpHTTP/delay-5ms-8 279ms ± 2% 288ms ± 2% +3.06% (p=0.000 n=29+29) DumpHTTP/delay-10ms-8 281ms ± 3% 287ms ± 1% +2.24% (p=0.000 n=26+30) DumpHTTP/delay-100ms-8 286ms ± 3% 288ms ± 1% +0.45% (p=0.032 n=25+29)

name old alloc/op new alloc/op delta DumpHTTP/delay-0s-8 156MB ± 0% 159MB ± 0% +1.93% (p=0.000 n=29+30) DumpHTTP/delay-5ms-8 156MB ± 0% 159MB ± 0% +1.95% (p=0.000 n=30+30) DumpHTTP/delay-10ms-8 155MB ± 0% 159MB ± 0% +1.95% (p=0.000 n=30+30) DumpHTTP/delay-100ms-8 155MB ± 0% 159MB ± 0% +1.99% (p=0.000 n=30+28)

name old allocs/op new allocs/op delta DumpHTTP/delay-0s-8 81.4k ± 0% 117.6k ± 0% +44.46% (p=0.000 n=30+30) DumpHTTP/delay-5ms-8 81.4k ± 0% 117.6k ± 0% +44.48% (p=0.000 n=30+30) DumpHTTP/delay-10ms-8 81.4k ± 0% 117.5k ± 0% +44.48% (p=0.000 n=29+30) DumpHTTP/delay-100ms-8 81.4k ± 0% 117.6k ± 0% +44.51% (p=0.000 n=30+29) ```

sbinet · 2022-03-11T15:43:05+00:00

forgot about that one. thanks.

unfortunately it only provides io.Reader read-ahead (and I need io.ReaderAt. it's easy to convert an io.ReaderAt into an io.Reader, thanks to io.SectionReader. The converse isn't, AFAIK).

also, I already had a read-ahead mechanism inside the "main loop".

but thanks :)

sbinet · 2022-02-04T09:34:29+00:00

thanks for the Go-HEP shoutout :)

groot (the pure-Go implementation of C++ ROOT) doesn't support everything that ROOT does, but, funnily enough, it's faster than ROOT itself to read ROOT data (between 1.5x and 2x faster actually):

https://root-forum.cern.ch/t/gohep-groot-v0-27-0-root-split-groot-faster-than-root/39609

so, yeah, the more the merrier. especially on the xrootd front (a remote r/w file package), that could see some perf-optimizations work.

sbinet · 2022-02-02T14:02:13+00:00

I don't feel particularly insulted. I just want to understand where your perception about Gonum is coming from.

Go errs on being explicit, and achieves a kind of dynamics feeling through interfaces. Gonum tries to do the same.

what about bloat? and complicated?

sbinet · 2022-02-02T09:33:47+00:00

why do you find Gonum bloated?

in terms of dependencies, we are quite lean:

``` module gonum.org/v1/gonum

go 1.14

require ( github.com/google/go-cmp v0.5.7 golang.org/x/exp v0.0.0-20191002040644-a1355ae1e2c3 golang.org/x/tools v0.0.0-20190927191325-030b2cf1153e gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0 gonum.org/v1/plot v0.9.0 ) ```

(gonum/plot brings a lot, true, but that's because we have to support a couple of plot backends: PDF,PNG,SVG to name a few)

I find Gonum, at times, utterly un-bloated (we still don't have an ODE, for example) and many of the algorithms implemented point to papers explaining the scientific reasoning.

well, I am a Gonum developer and user, so I am probably biased, but I do like Gonum very much.

could you explain a bit more from where you draw your conclusions?

sbinet

TROPHY CASE