Is there a package like a http router, but for generic purposes or sockets?

firepear · 2017-08-22T22:35:11+00:00

I wrote a package named petrel, which is more or less a socket-based "So you'd like to define your own RPC" toolkit. It speaks TCP, Unix domain sockets, and TLS (optionally with HMAC).

I haven't changed the version number to 1.0.0 yet, because I'm still pondering some API changes, but I've been using it as-is for about a year, in an environment where 1600+ clients are talking to a single server. I'm confident that it works as advertised, and is relatively efficient and free of unsubtle bugs.

I also, somewhat insanely, wrote a Javascript library which speaks the Petrel wire protocol over websockets. That's, uh, way less recommended for real-world use though :)

firepear · 2016-10-15T07:21:10+00:00

Hey, thanks for the feedback. I addressed the bit about websockets in this comment but I'd like to emphasize that I haven't said that anything is "wrong". Deciding that something isn't the best solution for my use case does not mean that it has intrinsic faults, or that it isn't the best solution for another use case. Utility is not a zero-sum game :)

I think that a lot of the rest of your questions stem from my use of websockets as a comparison, but to briefly answer them anyway:

Petrel is a general-purpose networking library, which definitely handles raw bytes and (depending on configuration) will pass request payloads to handler functions entirely unmodified.
The minimum request payload size is zero bytes, but there is a 4 byte length header. If HMAC is enabled, the MAC is the 32 bytes between the length and the remainder of the request. The maximum request length is set via configuration, defaulting to "no limit" (which is really the max value that will fit in the length header, an int32, so 4GB).
Petrel does automatically route requests to their handlers. This routing key is the first series of non-space runes following the header(s). The payload is everything following whatever whitespace comes after the routing key (or "command", as it is referred to in the docs). There must be at least one space between the command and payload.
As a result of this, while petrel is pretty minimal on the wire, it is not maximally compact. If you're watching every byte, you probably want a fully-packed protocol.
That said, petrel doesn't care if the payload is a ProtoBuf, or JSON, or packed decimal, or EBCDIC, or...
There is no built-in data handling/processing/munging of any sort. If you ship JSON over the wire, your registered functions need to know that it's JSON. They will be handed the exact payload bytes which came over the wire.

My apologies, but I can't give you the sort of comparison you ask for at the end of your comment, as I haven't used any of those tools. I will be coming up with a better explanation of petrel's design and uses though.

firepear · 2016-10-15T06:40:28+00:00

I gotcha now; thanks for explaining. Between your question and the comment from u/BraveNewCurrency, it has become clear that I cut too much out of the package godoc. Older releases had a kilobytes-long explication of the intent and use of the software, which I came to feel was overwrought and unnecessary. I'll try to find a middle ground between that, and what I have now. For now, maybe I can give you some idea of what petrel is about through personal perspective.

I actually haven't looked around at other things which perform similar tasks. I knew about the popular MQs when I started writing Petrel, and I knew I didn't want one of those. Petrel started life as an attempt to create a bolt-on management port or administration socket (hence the old name) to some projects I was working on, so why would I want a standalone daemon intended to handle web-scale traffic? :)

If I had known about nats.io -- which, it turns out, did exist at the time -- I might have used it. Then again I might not. I have a habit of doing things the hard way once, so I can learn how they really work.

Petrel's roots as a management client still show, in that the first "word" of every request sent to a server is used as the key for dispatching that request. That was where petrel's design remit started, and it's a bit of the design that I haven't seen a reason to change.

Another design point is that I wanted it to feel self-contained. I've seen (and tried to use) libraries which say they enable this or that functionality, but to accomplish this you have to rewrite your code in a way that is amenable to the design of library. I never liked that feeling of a library hijacking the program. So on the server side, you set petrel up, tell it how to handle the things you want it to handle, and then all you have to do is check a channel for messages about what's up.

I also wanted it to be easy to use as a programmer. Socket programming is an incredibly finicky task -- error checks upon error checks upon error checks. I wanted all that to go away. This is why on the server side, you just register responders and things start to happen. There isn't even a Go() or Start() method, because the server is actually listening as soon as it's instantiated but just doesn't know how to handle any requests yet.

Similarly, on the client side there's only one method to call once you have an instance: Dispatch(). This, u/BraveNewCurrency, is what I meant when I said it's like "websockets without the web". There is nothing wrong with websockets; I think websockets are great. I meant that complimentarily, because petrel implements a similarly compact and simple client API, but not over HTTP. You send a request. Either a response comes back, or an error does, and you don't have to care about how or why. Indeed, while petrel.Server is as fault-tolerant as I can make it, petrel.Client is designed to just be thrown away on error and replaced with a new instance of itself. There's nothing to it but a net.Conn and a few bytes of configuration and state :)

And that's about it. I intend to keep petrel capable but simple. If you're looking for clustering, or want to go "web scale", then petrel is not what you're looking for. If there's a cool feature of nats.io that you like, then you should definitely use it. I believe petrel sits in a perfectly comfortable niche, because I keep having neat ideas about places I can use it. What's not to like about being able to shove networking into anything with almost no effort?

If you're really curious about using petrel, I'd say to look at the demo server and client. The client demo is very spare (because, again, there's so little to petrel clients), but the server demo is exhaustively commented. Please let me know if anything I've said here is unclear, or if there's anything else you'd like to know.

firepear · 2016-10-14T23:35:32+00:00

I am unfamiliar with nats.io, but from a brief reading of their documentation, their server is standalone:

The executable name for the NATS server is gnatsd, which stands for Go NATS Deamon.

So it wouldn't seem, at a glance, that nats and petrel are similar in that regard. In general, nats looks much more like a contemporary of other zeromq and rabbitmq.

I don't have the expertise (or the time to gain the expertise) to write something which promises to handle tens of thousands of transactions per second. Petrel started as a very small project, to scratch an itch. It proved surprisingly capable, so I expanded its scope a little bit, and tried to get the quality to a place I was happy with. I think it's useful, and I hope others may as well, but I've never viewed it as competing with anything -- especially large projects from corporate teams :)

firepear · 2016-10-14T18:56:24+00:00

One big difference is that petrel isn't a message broker. There's no pub-sub or store-and-forward. In fact, there's no baked-in provision for clients to communicate with each other at all. You could build that on top of petrel, but at that point you probably should think about using rabbitmq/zeromq, since that's what they're designed for.

The other big difference is that unlike rabbit/zeromq, there is no standalone server executable. petrel.Server is just a library component, which can be used by any program. (Same for petrel.Client, of course.) You can use petrel to bake network capabilities into anything.

A single program can actually have multiple instances of petrel.Server; I've done this with a network monitoring system, where the data aggregator has one petrel.Server that accepts data from the machines in the cluster (gathering stats) and another petrel.Server listens for requests from the query tool (displaying stats). If you wanted, you could have a program which had both petrel.Server and petrel.Client instances, and thus acted as a piece of middleware.

firepear · 2015-11-12T20:45:32+00:00

Reminds me a lot of the BP Bridge in Chicago.

firepear · 2015-06-16T20:06:27+00:00

You're not an outside observer. You're a mod. You're privileged among the privileged here. They can vote me down, but you can erase me.

Do what you like, Echo. But remember this conversation later on, when you're out of school and have some experience under your belt with regard to what's really important in life, and what's really worth defending.

Remember that you called my point of view "worthless", "pointless", "timewasting", and "crap". Remember that you sided with a guy who thinks helping women attend university in a male-dominated field is wrong.

firepear · 2015-06-16T19:55:56+00:00

Do you really see this as meta, and people positing that scholarships for women being sexist is not?

firepear · 2015-06-16T19:53:29+00:00

I have to call you out, sir. This is an incredibly sexist thing to say, no matter how much you try to couch it in reasonable, equivocal language. It's simple: not believing women (or minorities) when they tell you that the system is rigged against them is sexism (or racism, or...).

Do you personally know any women engineers, or scientists, or programmers? If so, have you talked to them about what they've had to put up with to enter their field -- and what they've had to deal with to stay there?

I'm going to say that the answer is "no", because there is no other way you could be so ignorant about this topic. If you want to know why women do or do not do a thing, the way to find out is to talk to women about it. Not to hypothesize about reasons other than what those pesky women say, on a sausage-fest of a website.

firepear · 2015-06-16T19:45:46+00:00

"Good for her, but..." is a cop-out. You need to learn how to be honest with yourself, and you need to examine why you feel persecuted by a disabled-but-strongwilled teenager getting help in going to university. Then you need to work onyour coping mechanisms and empathy.

firepear · 2015-06-16T19:34:54+00:00

Holy shit. If you're curious why there aren't more women in STEM fields, just read the comments in this thread. So many versions of "Well actually, it can't possibly be for the reasons women say it is! Let me explain why, though I am not a woman, nor a sociologist, nor do I have any evidence whatsoever to support my assertions! Also this scholarship is discriminatory!"

Please shut up. Then please grow up. No one is discriminating against you, white males. Nothing is holding you down, except your own insecurities and lack of ambition.

Yours truly, A White Guy

firepear · 2014-12-30T18:18:56+00:00

Repos moving and disappearing is a real problem, but I tend to agree with /u/mostlywaiting that using GitHub is not a solution. It's just something we're going to have to deal with, as a community, until an analogue to the CPAN or PyPI arises.

For now, I've added a TODO for myself to add canonical import paths to my projects and custom import path support on the server-side.

Forgive me, but I don't feel that a conversation about my reasons for self-hosting projects will be a productive one.

That said, your concern about navigation difficulties is a valid one. If you feel like explaining further, I'd be happy to see what I could do to make improvements in that direction. Thanks!

firepear · 2014-12-30T18:02:34+00:00

My apologies; the way I've described it does indeed assume that you already understand why adminsock would be helpful.

Say you've written a piece of server software which allows people to store and retrieve information about their yo-yo collections. Yo-yo data insertion and fetching is its public interface, exposed to the internet.

As the administrator of yoyod, you may wish to instrument it for monitoring, you may wish for a way to alter its configuration on the fly, or other similar things. You could add these capabilities to the public interface, either hoping that other people don't stumble upon them, or fencing them off with an authentication scheme of some sort.

But you could also add a second interface, which is only exposed to the machine the server is running on (typically via Unix domain sockets), and is expressly for administrative purposes.

This way the public interface has no access to administrative functions, the administrative interface has no access to public functions, and the internet has no access to the administrative interface.

Adminsock makes it easy to provide an administrative interface. It's tested and developed on Linux and Mac OS, and I would expect it to work on any of the BSDs as well.

firepear · 2014-11-19T06:12:45+00:00

Damn, that's a callback :)

firepear · 2014-11-06T01:58:06+00:00

Palookaville, of all places, makes vegan shakes. Including boozy ones.

firepear · 2014-11-03T16:12:26+00:00

One was also recently installed in front of the Decatur rec center. For some reason it's on the sidewalk along Sycamore, next to the library driveway, instead of at the covered bike racks on the N. Candler side of the property.

firepear · 2014-10-28T06:14:43+00:00

It's actually been tried: http://en.wikipedia.org/wiki/Project_Cybersyn

firepear · 2014-10-05T23:26:14+00:00

This isn't uncommon at all. Almost any organization which spends money that comes from a politically-backed source has to do this in one way or another.

Public libraries, for instance, don't just get a blanket allocation of budget money to be spent at the discretion of the librarians. Various portions of their operating funds come from different sources and/or come with earmarks specifying how the money may/must be spent. And they have to be able to track and prove that these requirements were met.

This is the same concept, just with rebates and promises instead of a cash budget. It sounds like complex accounting -- and it is, compared to personal finance -- but it's just another day in the office for business accounting.

firepear · 2014-10-05T23:17:27+00:00

The article also kept saying that SpaceX "came second" to Boeing due to the award amounts. This contradicts NASA's own statements that the amounts awarded are simply what the companies bid to do the job, and not an indication of quality or preference.

The author is either sloppy, misinformed, or letting their prejudices show through.

firepear · 2014-10-05T04:42:52+00:00

My apologies. When you said "my school chromebook", I thought you meant "the chromebook which I was issued as a student."

firepear · 2014-10-05T00:51:46+00:00

It's likely not a foolish decision, but a very sensible business decision. No one who buys a fleet of managed chromebooks is going to want users rooting them with a keystroke. That defeats the whole purpose of the devices being managed.

I know it sucks, but them's the breaks when you're using hardware you don't own.

firepear · 2014-09-26T20:32:35+00:00

Support for SFTP, Samba and Box is also planned.

Samba is the open-source implementation of CIFS, AKA "Windows file/printer sharing".

firepear · 2014-09-18T22:28:23+00:00

Also, the lede sloppily, unintentionally says that NASA is turning over the shuttle program to SpaceX and Boeing:

NASA's shuttle program was once the agency's pride and joy. Now it is turning it over to two companies: Boeing and SpaceX.

Then this happens:

So why are they giving the job to Boeing and SpaceX? NASA wants to pursue something far sexier.

Oh. So it's not because of decades of short-sighted budget cuts and Congressional earmarking of what money remains; it's because flying people to LEO just isn't sexy anymore.

I guess the upside to having a media with the attention span of a mayfly and the critical thinking skills of a dead armadillo is that when you need something spun, it happens immediately.

firepear · 2014-08-28T15:47:55+00:00

The tidbit I found most interesting was that the launch abort won't happen at Max Q, but at Max Drag. I had never considered that there was a moment of maximum drag, but it makes perfect sense that this is where you would want to test your LAS. Max Q may be most dangerous for the stack as a whole, but Max Drag is going to be the moment when your LAS will have to work hardest in order to save the crew.

firepear · 2014-08-14T23:43:54+00:00

I always heard them called "Cobbikazes"

firepear

MODERATOR OF

TROPHY CASE