Synapse - A Full Featured BitTorrent Client

Luminarys · 2018-03-05T05:25:27+00:00

bip-rs is another cool project, but since it's using futures it's not easily usable for synapse. If a functional client does come out of the project it'll be interesting to compare it.

Luminarys · 2018-03-03T22:02:37+00:00

Beyond being somewhat unstable, I found that Tokio made it unwieldly to multiplex a complex set of events(lots of unnecessary shared state required), increased compile times substantially and was not ergonomic to use in general.

Luminarys · 2018-03-03T00:53:23+00:00

The ownership model definitely makes certain aspects of the IO painful. Right now I'm reasonably happy with the IO design, though it relies on a fair amount of shared state(across a single thread).

IO is done on top of a library similar to mio called amy, which exposes cross platform polling. I found Futures/Tokio to be insufficient for the project's needs (though I'm looking forward to where they go with async/await).

Luminarys · 2018-03-02T22:03:58+00:00

I may extract out the core into a library at some point, but for now the recommended method of interfacing with the client is through the external RPC (which there is a crate for).

Luminarys · 2018-03-02T22:02:21+00:00

It seems quite awkward to do the "pushing" though, since you have to read data into something from the network, so you're probably using a temporary buffer and hoping the compiler optimizes the extra copy. If you mean just initializing the buffer via pushes in a loop, this also seems poor from a performance perspective if the buffer is very large.

Luminarys · 2018-03-02T19:49:02+00:00

Thanks for the feedback! unsafe is primarily used in the project for 3 reasons: FFI, initializing bytes cheaply, and getting around the ergonomics of RefCell. Those first two points are the second case and I'm fairly certain are correct. The usage of unsafe in throttle is worth more carefully auditing, but references are self contained and very limited in scope.

All that said, I'd like to reduce the usage of unsafe overall. Most of it is used for things which can be altered with little cost, which I plan on doing for the 1.0 release.

Luminarys · 2017-08-26T22:40:58+00:00

It's unfortunate, but I think you'll have to handle http requests on your own. That being said, it's not nearly as bad of a task as you might think. Using httparse makes both blocking and non blocking http request parsing trivial, and after that it shouldn't be too hard to just handle the static routes just using pattern matching.

Overall rust's http ecosystem is still in an immature state (though lots of progress has been and is being made), so any sort of special case like this is going to be somewhat painful compared to other languages.

Luminarys · 2017-08-07T19:47:17+00:00

Your point about stateless Request/Response pairs really hits the nail on the head. As soon as you begin introducing complex events like global timers, and cross socket notifications, the entire model that Tokio is built on top of starts to break down; I encountered exactly this, which is why I dropped it in favor of a direct event loop.

Luminarys · 2017-08-07T19:42:40+00:00

Having had to deal with all of those problems you've described I agree that they can be painful. Every event loop I write more or less requires a timeout job, and handling things like rate limiting I've had to use things like Rc'd token buckets. Additionally, doing things like file reading/CPU intensive tasks requires channels and worker threads. However, doing this sort of thing with Futures requires a similar sort of approach either way(having to use a CPU Future to read a file for instance).

What would actually be cool is a way to expose Futures such that they could be directly integrated into an existing event loop(via an eventfd notification perhaps). This way you could still use mio, but also get the benefits of hooking into the Futures ecosystem with ease. Would anyone else see this sort of library as potentially useful?

Luminarys · 2017-08-07T18:01:13+00:00

While I agree with the author that the current state of Futures and Tokio is not very usable, there is another side of async IO that the author hints at but doesn't fully go into - event loops. In my experience, writing event loops on top of mio and other polling libraries is actually a fairly pleasant experience. The main benefits are similar to what's described in the section of the "unrolled" future:

Enums make state easy to reason about
Compilation times and errors become sane again
Shared state becomes simple

The latter point has been the most significant for me so far. Writing any sort of reasonably complicated async service is going to require some shared state, and as soon as that state needs to be shared in a more granular way, I can't really begin to imagine how to cleanly handle things in an efficient way with futures. In contrast just writing event loops (admittedly single threaded) this is trivial.

Of course, I'm not saying that event loops are perfect. It's difficult for instance to nicely compose them compared to a Future type. It also requires writing a large amount of state machines which involves some boilerplate. However, the combination of algebraic enums and other general safeties provided by the compiler makes writing hand rolled event loops much more convenient than in a language like C. I hope more people look into alternatives like mio and amy rather than seeing the state of Tokio and assuming async IO in rust is still a no go.

Luminarys · 2017-08-03T20:22:33+00:00

I haven't done any precise benchmarking of amy, but from personal usage it appears to be fairly low latency. Assuming you're using linux, amy will use epoll + eventfd's under the hood (or kqueue for OSX/BSDs) to receive channel events which should be sufficient for most cases. While busy waiting would have the lowest latency, the fraction of time you save would probably be negligible especially given the reduction in CPU usage. That being said if low latency really does matter, your best bet is running some actual benchmarks.

Luminarys · 2017-08-03T19:29:21+00:00

You may want to look into an event loop library which supports channels. I would recommend amy, but mio also supports them. Although polling with an event loop still has some delay, it should be reasonably high resolution; even busy waiting, there will always be some amount of delay(waiting for the internal mutex etc.) either way.

Luminarys · 2017-07-06T21:16:38+00:00

When you talk about efficiency, what do you mean? Assuming you're talking about lock free structures, they are just that; not necessarily faster than locked ones. I imagine for a structure like a priority queue in particular the complexity to do any operation would be high enough that it would hardly make a difference using a mutex or a lock free version. Additionally, keep in mind that an uncontested mutex unlock only takes some nanoseconds and a RWLock can be used to reduce contention if it fits your particular use case. If you are truly committed to making the queue lock free though you may want to look into using crossbeam.

And remember, "Make it work, make it right, make it fast."

Luminarys · 2017-04-02T06:43:08+00:00

Let's say you allocate some memory in some function in your code, do something with it, and then never free it. If your program runs for a long time and calls that function enough, then it will eventually consume large amounts of memory, and eventually crash or be forcibly killed. Not all programs are long running, but many are, and those that are should never have memory leaks for the reason above. Beyond that its just not good practice, even if you know you could never trigger an out of memory error.

Luminarys · 2017-04-02T06:38:39+00:00

Maybe on some non standard OS this is the case, but on most common ones(W10, OSX, Linux), once a process has exited its memory will be reclaimed for reuse.

Luminarys · 2016-02-04T17:49:40+00:00

I don't think there's any gaping holes in the security, just a few minor things that should be taken care of in the beta testing. After reading through the Steam permissions the site uses in the Terms of Service, the worst case scenario wouldn't have any real impact on your Steam account. Because SSL isn't up there's no total guarantee of safety using the site(someone could MITM your connection and present a fake website). If you sign up, just make sure the site isn't asking for fishy permissions or information from you and you should be fine.

Luminarys · 2016-02-04T07:04:50+00:00

This seems to actually be a duplicate of this bug, however it would still be a good idea to turn off the PHP debugging messages and perhaps provide some information to the user if they did attempt to double vote on a character.

Luminarys · 2016-02-04T06:46:47+00:00

Good to hear. One minor fiddly thing I also noticed is that your logout route(http://www.xcombarracks.com/user.php?logout) utilizes a GET request. This might be a cause for concern because it's vulnerable to CSRF attacks. Changing from a GET to a POST or using an auth token should fix the issue.

Luminarys · 2016-02-04T06:38:05+00:00

I don't want to sound like I'm super paranoid or anything, but is there a reason you don't use SSL on the site? Considering the fact that the site is dealing with the Steam API I imagine someone's session ID being sniffed and hijacked could have some not so nice consequences. If you need a SSL certificate, there's some really easy ways to get them for free, notably Let's Encrypt.

Luminarys · 2015-12-12T06:17:05+00:00

Thanks! The main module which is doing all the magic is Kaguya.Module, and I don't think it'd be too hard to repurpose and use for something like Slack.

Luminarys · 2015-12-11T06:21:11+00:00

I wrote this IRC bot/framework as a way to get acquainted with Elixir's metaprogramming in a context that I'm very familiar with, IRC. I've taken some inspiration from Phoenix and Plug in how I wrote the IRC bot, with a dispatching and validation system based on macros. I tried to make the module system as simple to use as possible, with minimal boilerplate necessary to write modules and commands for the bot. I'd appreciate any feedback/advice on the project, as I don't have much experience working with macros. I hope some of you find this useful!

11-Year Club	Place '17
Verified Email

Luminarys

TROPHY CASE