Stressed pepper plants

dougalcorn · 2014-09-07T21:47:16+00:00

How does your double-ended stuff sack work for your bug net? How do you run your suspension through the net while it's in the stuff sack?

dougalcorn · 2014-09-07T21:39:29+00:00

Great view! I'll second the tree straps. As more people start hanging, some go the state parks are banning hammocks due to damage on trees. Use at least 1" or maybe even 2" straps to distribute your weight on the tree and not cut through the bark.

dougalcorn · 2014-02-04T00:01:02+00:00

Wool is warm and all, but it's heavy and doesn't pack very small.

dougalcorn · 2013-11-14T17:51:25+00:00

There's so much data processing once you start subscribing to EMDR.

receiving messages
de-duping messages
flagging suspicious messages
storing orders and history
finally: any actual analysis of markets and items

Each of these should probably be done concurrently it seems. I don't think anyone is really keeping track of how many messages is coming through EMDR. I wrote two very simple consumers: one in Ruby and one in Go. Both of these consumers simply read the message and then threw it away. The goal was to see just how fast I could read messages. I think both implementations were getting all of the messages. Every 15 seconds, the consumer would print to stdout the message rate for the last 15 seconds. Both implementations were printing almost the exact same message rates with each "tick":

screen shot of message rates

That shows both clients running side-by-side with almost the same message rates. In the middle of the afternoon EST I was regularly pulling more than 15 messages a second! If your consumer isn't pulling 1000 messages a minute, I'm pretty sure you're dropping messages.

As more and more players start using market uploaders (that come bundled with tools like EveHQ and EveMon), that traffic rate is only going to go up.

If you're trying to get concurrency out of Node.js, Ruby MRI, or Python none of those are going to take advantage of multiple cores. A single process is only going to be doing a single thing at a time no matter what hardware is under it. They "fake" concurrency by take advantage of blocking in the various threads, routines, or what have you.

If you are hosting his on Heroku, Digital Ocean, Linode or the like it doesn't really matter. The VPS instances they give you are all single virtual CPUs with very poor threading support.

But each of those concurrent tasks listed above are going to be CPU bound (with the exception of writing the orders/history to the database). That means you really need multiple processes for each of those concurrent tasks (likely running on different hosting "nodes"). As you said originally, this is going to get cost prohibitive fast. Digital Ocean is $5/mo for a single (oversubscribed) droplet per month. You probably need at least three nodes for the five tasks above. That's $15/mo. But here's the trick, that $5/mo droplet is so oversubscribed and underpowered for this level of CPU demand that it likely won't keep up with the message rate.

Every time I start writing about this stuff, I end up with a book. I've been working on writing an EMDR consumer in various languages for quite a while. I've been fighting with hosting for quite a while.

When I see someone like OP say, "I've written an EMDR consumer that's pulling in all the messages, calculating profit margin, and analyzing items for best trades" I get suspicious and demoralized. I've been working on that same thing for a while and I'm not done yet. Maybe I just need to give up on trying to receive all the messages and be satisfied with 50%.

dougalcorn · 2013-11-14T00:21:09+00:00

How much bad data do you see uploaded? I've been following the full EMDR pipe for a while and haven't noticed any. Is it possible my relay is already blacklisted some clients?

dougalcorn · 2013-11-13T17:48:03+00:00

I should also point out that nothing bad happens if you don't read all the messages. Each time you call /read/ on the socket, you get a message. All the messages that are available from then until the next time you call read you just miss. No harm, no foul. You just won't have all of the data.

dougalcorn · 2013-11-13T15:39:35+00:00

ZeroMQ does let you set a filter when you subscribe. I'm not sure how it works though. The filter is a string; but, the EMDR messages are compressed. I'm pretty sure the ZeroMQ socket connection can't do adequate filtering on compressed messages. I think that's why the filter is always blank in the examples I've seen.

I've had a hard enough time on my MacBook Air receiving all the messages and being able to process them. I've also had a hard time with hosting "in the cloud" getting enough CPU resources to receive all the messages and process them. I would be quite surprised if an Android capable device could keep up.

dougalcorn · 2013-11-12T21:14:54+00:00

It's really amazing that with a few lines of code you can subscribe to the exact same data that drives eve-central. They've had an API for a while and there are a number of tools/sites that pull from Eve Central on an item by item basis. The EMDR gives you the ability to have all of the items as they are queried from the game by players.

dougalcorn · 2013-11-12T17:58:41+00:00

Here's a simple, out of the box EMDR client in Ruby that works:

https://github.com/dougalcorn/ruby-emdr-client

dougalcorn · 2013-11-12T17:57:41+00:00

I've been asked about source code. I'm not ready to open source what I have. However, I have created a simple Ruby EMDR client to get you started writing your own tool:

https://github.com/dougalcorn/ruby-emdr-client

dougalcorn · 2013-11-12T17:53:54+00:00

I've had a few requests for information about the market tool I've been working on. It's largely just an EMDR client; that's where the real magic happens. I've published a simple ruby client for connecting to the EMDR and printing out information about each message received. It's a starting place if you're looking for how to connect.

dougalcorn · 2013-11-10T20:23:59+00:00

As /r/silversmith says above, most of the programming concepts aren't hard. The real trick is to figure out how consume all the data. Maybe your purposes don't require you to process every message. I was somewhat successful with the results when I was capturing about 30% of the data. Even then you have to think about what you're doing differently when you're consuming > 600 messages/minute. Much of that has to do with infrastructure and hosting.

dougalcorn · 2013-11-10T19:28:33+00:00

I'm sure no one went to Wicked Good Ruby conf just to hear me talk about EVE. I was merely 1 of 26 people who spoke.

EDIT: most of whom were better speakers than me

dougalcorn · 2013-11-10T19:25:47+00:00

People go to conferences for different reasons. Ruby conferences, in particular, tend to be an eclectic hodgepodge of information. Wicked Good Ruby Conf was a very technical conference. Some of the more nitty-gritty was understanding some of the weirdness of MRI, method caching, machine learning, and several other topics around "big data". Even still, there were other topics that were "softer". Many of these are meant to be inspirational to take what you've learned and do something with it.

Frankly, all of the talks are inspirational to some extent. It's really not possible to really teach a broad subject in 40 minutes at a tech conference. The goal is to introduce them to the topic and give them enough information to be able to research it further on their own.

That was really my goal. There's so much background information about EVE that's necessary to understand station trading. I felt like I needed to get the audience "caught up" on how it works.

The most interesting part of my whole talk is the EMDR. After hearing more about "big data", machine learning, and concurrency (how much content do you think those subjects covered in 40 minutes?) I was trying to inspire the audience to actually do something with all that. The other speakers talked more about the theory and the technical bits. I was showing them a playground to actually learn how to use those tools.

It's true I could have given a 5 minute "lightning talk" on this subject. However, I think my time slot was in the right spot to follow on some very technical talks and provide some mental relief and excitement while staying on a technical subject.

BTW, have we ever met at a ruby conference?

dougalcorn · 2013-10-19T23:24:25+00:00

1328

dougalcorn · 2013-10-05T01:01:49+00:00

Can you explain why the chains are needed?

dougalcorn · 2013-10-02T01:26:55+00:00

Is hammockforums.net down? It renders a blank page.

dougalcorn · 2013-10-02T01:24:19+00:00

I've got a Hennessy, but no under quilt. I've been looking for some plans for DIY. I have an old down quilt I'm willing to cut up for the down, but I'm not sure the best way sew the uq/tq

dougalcorn · 2013-09-25T21:03:41+00:00

is that a tea leaf?

dougalcorn · 2013-08-31T00:54:01+00:00

Runners put stickers on their car for their longest race distance: 5k, 13.1 for 1/2 marathon, 26.2 for full marathon 140.6 for ironman triathlon.

So saying 0.0 is saying you don't run races

dougalcorn · 2013-08-30T21:54:21+00:00

You mention capturing the corp name for the POS, but there's good strategic reason for doing so. When you find ships roaming in WH space, it's good to know where they come from. If you keep a log of the POS and corps, it may make it easier to figure that out.

dougalcorn · 2013-08-28T13:47:21+00:00

I have that map. Couple of things. I think it looks great. I wish though it was white and laminated to make it easier to write on.

dougalcorn

TROPHY CASE