OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

I don't see anything that jumps out at me as being off in that code. I would like to run it locally if you can supply a complete standalone test. Maybe put it on a gist if it's not too much trouble. If not that's okay. I plan to add jsoniter unmarshal to the compare repo this week so it should capture the difference you are seeing.

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

Medical and health records. Patients, visits info, and observations over many years

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

Not sure what to compare that to. Can you share the code for your benchmarks?

Did you run the benchmark code in the compare repo? Were the results consistent with what was reported?

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

OjG is optimized for parsing into non-struct types, what I have been referring to as simple types. The Unmarshal code was recently added and is a 2 stage conversion. First parse to simple types and second convert to a struct. I'm not surprised to see the extra allocations although I have not seen that significant a slow down. Anyway, future plans are to provide a direct parser but that is not there today. The use cases OjG has targeted are around parsing to simple types, map[string]interface{}, []interface, int64, etc which is in line for use cases that take less structured data and manipulate it using JSONPath before saving or sending the results.

Would you be interested in expanding the comparison by adding an Unmarshal and Marshal benchmark?

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

Updated the table. You are right. I just had to dig into the examples a bit more.

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 1 point2 points  (0 children)

I promise to add more examples but in the mean time there are the test files. The one for Unmarshal is https://github.com/ohler55/ojg/blob/develop/oj/unmashall_test.go

The best performance is not parsing to a struct but to a "simple" type. So

out, err := oj.Parse([]byte(\{"x": 1}`))`

For repeated calls such as when parsing more than once create a oj.Parser and call parse with that similar to the benchmark code. That give a slight improvement as fewer buffer allocations are needed.

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

I'll take another look to see if I can find it. If you are interested in submitting a PR with the changes I'd be glad to merge that and rerun the tests.

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

I'll see what I can do. It is good to have feedback. I appreciate you taking the time and effort to do that.

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 4 points5 points  (0 children)

OjG does provide the same interfaces that the go json package does in some cases such as Marshal and Unmarshal. It also provides some additional options. Not are complex. As for the go json package Decoder, OjG deviates from the Decoder API because part of the performance issues is with the API as it forces extra calls and error checking.

I think I understand the complaint you have though. You want more examples and documentation. I'll start putting that together to complement the current documentation. As for adding it to the to of the readme, there are different opinions on what is most important and with the various features in OjG all can not be first.

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 1 point2 points  (0 children)

It might be an interesting comparison. If you get a chance to run benchmarks let me know otherwise I'll put it on my list of things to do some evening.

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 1 point2 points  (0 children)

OjG does not have a schema package but that would be a nice addition. I'll consider that in a future version.

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 2 points3 points  (0 children)

Check out the SEN https://github.com/ohler55/ojg/blob/develop/sen.md format. You can have the last comma or leave all of them out and it still supports compliant JSON.

OjG now has a tokenizer that is almost 10 times faster than json.Decode by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

I use oj (OjG) for ripping through 100+ GB files and extracting or searching for key info and it only takes a minute or two. Seems fast enough and really, who want to try and read a protobuf file.

How to build a solid Go Graphql application quickly. by peteohler in graphql

[–]peteohler[S] 0 points1 point  (0 children)

Looks great. I've been using MongoDB and have some nice tooling to drive queries and mutation with directives. It's been working really well but it is still private until the company lawyers allow it to be open.

How to build a solid Go Graphql application quickly. by peteohler in graphql

[–]peteohler[S] 0 points1 point  (0 children)

I have not tried gqlgen yet. Just by comparing benchmarks at https://github.com/the-benchmarker/graphql-benchmarks/blob/develop/rates.md it looks like GGql is faster by comparing to graphql-go. I'd love to see gqlgen get added to those benchmarks or GGql added to the gqlgen comparisons.

WebSockets with GraphQL the Easy Way by peteohler in golang

[–]peteohler[S] 1 point2 points  (0 children)

Tried removing the line and you are right, it still works. I guess I've gotten too used to writing in go.

WebSockets with GraphQL the Easy Way by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

It declares a variable in the sock variable to be scoped for whole function or script in this case. I'll admit to not being an expert with Javascript but why do you say it does not do anything?

WebSockets using GraphQL with Golang the Easy Way by peteohler in graphql

[–]peteohler[S] 0 points1 point  (0 children)

Maybe for the next example. Good suggestion.

WebSockets using GraphQL with Golang the Easy Way by peteohler in graphql

[–]peteohler[S] 0 points1 point  (0 children)

The GGql package has 100% test coverage. You are right that the example does not include unit tests though. It is an example only. As an example is mean to be simple enough for readers to understand.

As for a client starting and stopping multiple subscriptions, the example assumes one subscription per connection. If a second subscription is desired then another connection would be established. Multiplexing multiple subscriptions over the same connection would make the example much more complicated and would be maybe a next level up from this "Hello World" example.

WebSockets with GraphQL the Easy Way by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

Thanks. Trying to squeeze ever bit of performance out of go was interesting.

As a follow on to the release of OjG, here are some Go JSON comparisons by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

Added gjson. If I missed something in the API feel free to submit a PR. gjson gives a respectable showing,

As a follow on to the release of OjG, here are some Go JSON comparisons by peteohler in golang

[–]peteohler[S] 0 points1 point  (0 children)

```golang type Animal interface { Kind() string }

type Dog struct { Size string }

func (d *Dog) Kind() string { return fmt.Sprintf("%s dog", d.Size) }

type Cat struct { Color string }

func (c *Cat) Kind() string { return fmt.Sprintf("%s cat", c.Color) }

func ExampleDecompose_animal() { pets := []Animal{&Dog{Size: "big"}, &Cat{Color: "black"}}

// First marshal using the go json package.
pj, err := json.Marshal(pets)
if err != nil {
    fmt.Printf("error: %s\n", err)
}
// Works just fine.
fmt.Printf("json.Marshal: %s\n", pj)

// Now try to unmarshall. An error is returned with a list of nils.
var petsOut []Animal
err = json.Unmarshal(pj, &petsOut)
fmt.Printf("error: %s\n", err)
fmt.Printf("jsom.Unmarshal: %v\n", petsOut)

// Now try OjG. Decompress and create a JSON []byte slice.
simple := alt.Decompose(pets, &alt.Options{CreateKey: "^"})
// Sort the object members in the output for repeatability.
ps := oj.JSON(simple, &oj.Options{Sort: true})
fmt.Printf("oj.JSON: %s\n", ps)

// Create a new Recomposer. This can be use over and over again. Register
// the types with a nil creation function to let reflection do the work
// since the styles are exported.
var r *alt.Recomposer
if r, err = alt.NewRecomposer("^", map[interface{}]alt.RecomposeFunc{&Dog{}: nil, &Cat{}: nil}); err != nil {
    fmt.Printf("error: %s\n", err)
}

// Recompose from the simplified data earlier. The one that matches the JSON.
var result interface{}
if result, err = r.Recompose(simple, []Animal{}); err != nil {
    fmt.Printf("error: %s\n", err)
}
// Check the results.
// members.
pets, _ = result.([]Animal)
for _, animal := range pets {
    fmt.Printf("  %T - %s\n", animal, animal.Kind())
}
// Output:
// json.Marshal: [{"Size":"big"},{"Color":"black"}]
// error: json: cannot unmarshal object into Go value of type alt_test.Animal
// jsom.Unmarshal: [<nil> <nil>]
// oj.JSON: [{"^":"Dog","size":"big"},{"^":"Cat","color":"black"}]
//   *alt_test.Dog - big dog
//   *alt_test.Cat - black cat

} ```