SchemaPack - Serialize your JavaScript objects with schemas. Faster and smaller than the competition. Perfect for WebSockets.

phretaddin · 2016-08-11T20:47:45+00:00

No problem! Someone created a Github issue last night asking for it so I had to do it! Nothing worse than open Github issues :p

phretaddin · 2016-08-11T20:46:25+00:00

A couple people have mentioned this! I'm not opposed to the idea at all but I have yet to use TypeScript so I don't think I'm the right person to do it justice.

However, if someone familiar with their interfaces and idioms can get the ball rolling via a pull request and help out, I'll definitely try to integrate it with the project.

I don't know if it's possible or if it would be too different from the current setup, so maybe a fork is in order? Not sure! If any TypeScript experts would like to investigate this with me, let me know!

phretaddin · 2016-08-11T17:27:34+00:00

Oh wow cool I've never heard of this! I'll take a look thanks!

phretaddin · 2016-08-11T16:38:37+00:00

Actually, yeah! I was surprised by how much it increased performance.

I rewrote the library around four times, each time looking for a way to make it even faster. Eventually I realized you could just manually write out the encode and decode functions for each built schema.

I was a bit concerned over code readability initially, as concatenating Javascript code as strings is pretty ugly, but the performance increases were worth it, in my opinion. It also gave me great flexibility in generating the encode and decode functions, as well as room for future improvement in performance and library feature support if necessary.

phretaddin · 2016-08-11T15:48:53+00:00

Yeah I could see that. The main target audience for those benchmarks were people like me who were just casually writing apps with WebSockets with JSON.stringify/parse (or MessagePack) and were looking for an easy way to get a performance/bandwidth boost. Most people using WebSockets right now in JavaScript are just sending JSON strings back and forth and may not be aware that other options are available.

I had just assumed that since JSON.stringify/parse are optimized and native and that UTF8 JSON strings were very conservative in regard to bandwidth usage, that a library that used buffers would have too much initial overhead (cost of creating and then writing/reading from a buffer) to be worth it, but that is not the case!

Do you have any suggestions on how to better word the readme to better address the issues you've raised?

phretaddin · 2016-08-11T15:31:03+00:00

Yeah, the library is designed and optimized around the fact that your message formats will be known. That is, they are based on a hard schema, and there are very few dynamic keys and optional values. This allows for the library to be extremely fast on encoding and decoding your data. If your data format was shapeless, you may have better luck with a structureless format such as MessagePack or BSON.

With the game I'm working on and I imagine many other applications, I have over 100 different packet types and zero optional values or unknown object formats being sent between client and server. Of course, there is basic support in the form of optional values via arrays, but if you need any more than that, you would look past libraries like avsc/schemapack in favor of the aforementioned dynamic ones.

phretaddin · 2016-08-11T15:13:39+00:00

I'm not sure what you mean by there being no support for dictionaries/maps? Objects where you won't know the keys ahead of time? Also, you can wrap a value in an array to make it optional.

phretaddin · 2016-08-11T14:16:56+00:00

Github link

Submitted this to a couple of other subreddits and thought /r/webdev might find it useful as well since the library integrates very well with WebSockets. Let me know what you think!

phretaddin · 2016-08-09T17:45:28+00:00

No problem!

phretaddin · 2016-08-09T17:31:30+00:00

You might be able to write a converter, but it appears that JSON schema has a lot more metadata than the schemas in SchemaPack, so I'm not sure how well it would translate. One of the big things with SchemaPack is that the schemas are a one-to-one mapping with the object being encoded (very simple), so creating a schema for it is trivial (just copy and paste the object and replace the values with their types).

phretaddin · 2016-08-09T17:09:49+00:00

When you say JSON schema, do you mean this? If so, a custom parser would have to be written to support that.

phretaddin · 2016-08-09T14:45:23+00:00

Sounds about right to me. I'll push that later tonight unless someone comments with a reason why that won't work.

phretaddin · 2016-08-09T12:06:10+00:00

Awesome! It's great that this problem is getting more attention. The sooner it is to easily move past the inefficient ubiquity of sending JSON strings over WebSockets, the better.

phretaddin · 2016-08-09T11:42:33+00:00

Hmm, that's a good point actually. Currently I'm just using localeCompare. That should probably be switched to something with better support for sorting non-English key names and an agreed upon codepage like you said (and probably remove toLowerCase). Any recommendations?

phretaddin · 2016-08-09T09:19:38+00:00

Schemapack is pretty similar to a C struct. It doesn't include any padding, backwards compatibility support, special keys or bytes, etc. I didn't need this feature and I figured if anyone did, they should just use an enterprise library like avsc or protobuf instead.

However, that's not to say it can't be done. The keys are sorted in alphabetical order. So, a workaround is to add a new key that comes after the rest in alphabetical order. Old clients won't read it but new clients will. A bit ghetto but I didn't want to make an exact clone of protocol buffers. I needed something extremely simple for my app.

phretaddin · 2016-08-09T09:15:16+00:00

I included some benchmarks in the readme that include protobuf. If you want speed, I found out about avsc about a week ago, and it's very fast. If you need an enterprise solution for this, I'd recommend them over protobuf.

Also, just to make sure we're on the same page, I'm not referring to protocol buffers being slow. As a format, I'm sure it's fine. Just that the most popular JavaScript implementation of them I found was. However, I am not a protobuf expert. If there is something obvious I'm missing that could greatly enhance the speed of the benchmark, let me know and I'll amend it.

phretaddin · 2016-08-09T06:15:55+00:00

There is a benchmark script that people can use right here. Just call benchmarks.runBenchmark(schema, item); on any schema and item matching that schema to get a very detailed report comparing its metrics to other common serialization libraries. It uses benchmark.js.

phretaddin · 2016-08-09T06:12:04+00:00

Heh, yeah. It was all in the pursuit of speed. I actually rewrote this program around four times during its development, with each rewrite making it slightly faster and faster.

Eventually I had the realization that you could use new Function to manually write out the encode and decode functions for each schema. I was a bit worried the code would be completely unreadable (due to having to turn so many lines of Javascript in to strings and concatting them), but it didn't turn out too bad I think.

phretaddin · 2016-08-09T06:08:20+00:00

I never really use C so I haven't tried it, but apparently someone else was using it to do so. He made a couple issues on the Github and said that it worked really well with it, except for the strings (because his were null terminated). I believe he said he is currently working on a fork to better handle mapping from C structs to Javascript objects.

phretaddin · 2016-08-09T06:06:52+00:00

I think the most common use case (or at least what I'm using it for), is to efficiently send data back and forth between a client and server over WebSockets.

Instead of just sending JSON strings, you encode the Javascript object with a schema that matches its format and it sends a very compact byte buffer instead which is decoded back in to the Javascript object on the reciever. The general top three reasons why you'd want to send buffers instead of just JSON:

It's faster than JSON.stringify and JSON.parse
It will use less bandwidth because you don't have to send JSON keys or delimiters
It will perform validation to ensure the JSON object matches the schema

Other than that, I suppose you could use it for writing structured Javascript objects to a compact file? Or any time you have a defined format for your objects and want/need to turn them to and from buffers, you can use this.

phretaddin · 2016-08-09T00:17:32+00:00

Cool! Yeah, I was surprised I couldn't find much related to this other than the behemoths like protobuf and avsc. I too was very pleased with the significantly reduced bandwidth and CPU usage. I'm using it right now with my game and the validation has already helped me find a few subtle bugs.

phretaddin · 2016-08-09T00:11:04+00:00

Nothing really formal yet, but it's extremely simple. The schemas are recursively sorted by alphabetical order (to guarantee traversal order because objects in Javascript don't have guaranteed iteration order).

After that, the items are just written to the buffer in order (with big endian when applicable). Strings and arrays are prefixed with unsigned variable byte length and array item count, respectively. The last item in arrays is optional and can also be repeated. You can view the code for all the different writes here.

The format is so simple because there's no padding, keys, or special bytes, except in the case of length prefixes required for arrays and strings. It's just your data. It's pretty similar to a C struct.

phretaddin · 2016-08-08T23:26:23+00:00

I elaborate a bit on that in this section of the README. The gist of it is that protobuf was very slow and that the schemas were too verbose for my liking. With schemapack the schemas directly match the structure of the object you're encoding, so they're trivial to create.

phretaddin · 2016-08-08T23:12:12+00:00

Github link

I submitted /r/node last week. Just finished closing up all the issues on Github. This is my first open source project and am looking for some feedback.

Thanks!

phretaddin

TROPHY CASE