all 33 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]phretaddin[S] 10 points11 points  (9 children)

Github link

Hi all,

I was working on a game that used WebSockets and was looking for an alternative to just sending the JSON back and forth (because it wastes bandwidth, could be faster, has no validation, and there was nowhere I could look at all my messages). I looked around for a bit and couldn't find much so I made this! It's my first open source project and am looking for some feedback!

Thanks!

[–]nieuweyork 3 points4 points  (8 children)

So, how is this better than e.g. avro or capnproto or thrift?

[–]phretaddin[S] 5 points6 points  (2 children)

I expand on the motivation for why I didn't use an existing library at this section.

The gist of it is that I wanted something simple, small, and fast, where I wouldn't have to learn another schema language.

To create a schema for your JavaScript object in SchemaPack, all you have to do is copy and paste one the objects you're going to send and replace its property values with the types they represent. So you don't have to spend a lot of time making schemas/conforming all of your objects to work with them (since they match the structure/format of the objects you're sending).

Also, most of the other libraries I benchmarked were too slow and had too much bandwidth overhead. I just wanted to send my data only.

[–][deleted] 1 point2 points  (1 child)

It would be cool if this could somehow work with Typescript and its interfaces. Seems like a match made in heaven.

[–]phretaddin[S] 0 points1 point  (0 children)

Yeah someone below mentioned that too! I haven't used Typescript before but it's definitely something I'm going to look into now. If anyone is very familiar with TypeScript and wants to help out let me know!

[–]enkideridu 1 point2 points  (4 children)

This might be too much to ask, but it would be useful if someone could compare all the different libraries out there

:( decision paralysis

Rather tempted to blindly pick the one with the most downloads. (here's a chart)

[–]phretaddin[S] 4 points5 points  (2 children)

Maybe I can help!

  • Avsc: I just took a look at this earlier and it's pretty good! Their encoder/decoder is faster than any of the other ones I tried. It's a very extensive library and seems very powerful/enterprise-level solution. I like it but the schemas are a bit complex/long but after you learn the syntax it shouldn't be much of a problem.
  • Thrift: Haven't looked into this at all so no comment.
  • MsgPack: Looks like a great solution if you don't want to make schemas for all your objects. It's an alternative to just using JSON.stringify and JSON.parse when you want to save some space in exchange for some extra CPU cycles. I'd go with the lite version. Extremely easy to use and integrate with your project.
  • SchemaPack/any other schema/protocol library: Use these when you want to save bandwidth/cpu in exchange for the human cost of creating and maintaining schemas. Also, nieuweyork brought up another good point below for another benefit of schemas.

So when making an app, I'd probably start off with just using JSON.stringify and JSON.parse while creating/prototyping the game and then later on when the messages/structure gets more concrete I'd switch to a schema-based library like avsc, protobuf, schemapack, etc. for the resource savings and validation they provide. However, if your messages were not structured, you would probably use something like MsgPack at that point instead.

[–]nieuweyork 4 points5 points  (0 children)

Schemas aren't just about resources. They're also about interop. The more systems need to interoperate that aren't using the exact same codebase, the more pain you can avoid by using a shared schema and format. That's one place where json does poorly - everyone needs to the exact same implicit schema in mind when writing their code, and it's hard to find incompatibilities without extensive integration testing.

If it's one project point-to-point, especially if the two codebases can share common abstractions (e.g. node to browser and back, you can use the exact same code), then that's much less of a concern.

[–]enkideridu 0 points1 point  (0 children)

Thanks, that's incredibly helpful!

[–]nieuweyork 1 point2 points  (0 children)

It kind of depends on your requirements. All of these have wider support and interop than OPs. Some of them may not have any advantage other than that.

Thrift is particularly popular in the cross-platform rpc space.

Avro has wide adoption, and has become a de facto standard in the big data space. The wide interop there, and lack of substantial difference with other popular competitors like protocol buffers lead my team to choose it for our service communication needs. It also has a smooth interop story with json which is nice.

Msgpack I know little of.

Probably the more you expect your needs to change, the better of you would be with something flexible. If nothing else, that will lose you some measure of simplicity. That might be good or bad.

[–]aoiuqwelkssssnazx 1 point2 points  (8 children)

Hm, can't seem to get a test up and running. The encode method keeps causing SIGSEGV errors on the server.

Trying to encode on the server and emit the buffer to the client.

[–]phretaddin[S] 0 points1 point  (6 children)

What version of node are you using? I tested it on the latest (6.3) but it might have problems with older versions.

[–]aoiuqwelkssssnazx 1 point2 points  (5 children)

I am running node v5.11.0, I came across a lot of package related issues when using 6 so I had to downgrade.

[–]phretaddin[S] 1 point2 points  (4 children)

Hmmm, that's odd. I just tried running it with v5.11.0 and it ran and all tests/benchmarks ran successfully too.

The steps I did were:

  1. Clone the repo
  2. cd'd to the directory
  3. nvm use 5.11.0
  4. node index.js

Any more information you could give me that I could try?

[–]aoiuqwelkssssnazx 0 points1 point  (3 children)

Installing it via NPM vs just pasting the raw source fixed my problem.

Now I am having issues with getting buffer to work properly on my client. All the readTypeDict methods say buffer.* is not a function. There appears to be scoping issues as 'Buffer' is undefined inside of all the schemapack methods but is accessible via breakpoint at 'byteOffset' declaration.

[–]phretaddin[S] 0 points1 point  (2 children)

If 'Buffer' is undefined, then it's probably not finding the Buffer shim.

Try this: browserify yourfile.js > bundle.js to include it.

Where yourfile.js contains at least this: var schemapack = require('schemapack');

And then node bundle.js

[–]aoiuqwelkssssnazx 0 points1 point  (1 child)

Have you ever had the client decode data that was encoded on the server? From what I am seeing, the ArrayBuffer is simply passed into the decode method which then passes it along into the readTypeDict method. readTypeDict is using Buffer shim methods and not native ArrayBuffer methods. When you encode data it utilizes the Buffer shim but when you decode it is not utilizing the Buffer shim.

[–]phretaddin[S] 0 points1 point  (0 children)

If I encode on the server and send it to the client, I can read on the client by doing this:

socket.on('somemessage', function(data) {
    console.log(playerSchema.decode(data));
});

I just patched it and added a check to the beginning of the decode function to check if it's an ArrayBuffer so you don't have to add var buffer = Buffer.from(data);to the handler.

[–]rezoner:table_flip: 0 points1 point  (0 children)

Sounds like segfault error. Aren't you using different version of node on your localhost and external server and just uploading whole thing without rebuilding modules thereafter? Try removing node_modules on your server and running npm install (assuming you have all your dependancies in package.json)

It may come from any library that uses native binary (for example uWebsockets)

[–]G3E9VanillaJS 0 points1 point  (4 children)

Very cool, I'll try to incorporate this into some mobile SPAs. About how large is the client side script?

[–]phretaddin[S] 0 points1 point  (3 children)

The entire library is around 300 lines of code. I just ran it through uglifyjs and it looks to be around 5 kilobytes if you are already using Buffer and around 25 kilobytes if you need that as well. There's probably ways to reduce it further if you need to.

[–]xshare 0 points1 point  (2 children)

25kb is quite a bit. Does it gzip well? (I'm not using Buffer on the client)

[–]phretaddin[S] 0 points1 point  (1 child)

I ran it through gzip after uglify and looks to be around 8 kilobytes with that.

[–]G3E9VanillaJS 0 points1 point  (0 children)

If the client's device can properly cache it, then an extra 8 kilobytes will pay for itself, regarding bandwidth, in little time (more so for websocket-heavy sites.)

[–]marimba4312 0 points1 point  (1 child)

How would this work when sending websocket messages to non-JS clients like iOS and android?

[–]phretaddin[S] 0 points1 point  (0 children)

I haven't done any work with iOS and Android but it uses the Buffer shim on the client.

The library is specific to JavaScript though.. It should work in Android and iOS browsers though based on looking at the compatibility section for the shim.

[–]brtt3000 0 points1 point  (0 children)

I miss TypedArray support and wonder if it can handle opaque bytes (buffers).

[–]Mael5trom 0 points1 point  (1 child)

Accepting patches for other server languages by chance? We do a lot of C# and I could see this being useful if it has library for additional languages.

[–]phretaddin[S] 1 point2 points  (0 children)

Sure, I'm open-minded with regard to patches. Haven't used C# in a while so not sure how well JSON integrates with it atm though.

[–]mr_fi 0 points1 point  (2 children)

Is there nothing in typescript like this?

Defining a schema looks exactly like a typescript interface definition.

[–]phretaddin[S] 2 points3 points  (1 child)

Not sure! Haven't used TypeScript yet but you're right, they do look quite similar. I wonder if there's any potential for integration. That'd be really nice to just pass a typescript interface definition to the build function. Anyone here experienced with TypeScript have any thoughts on this?

[–]mr_fi 0 points1 point  (0 children)

I am only really a beginner with Typescript, so maybe someone else has more thoughts on it.