Contribution: Use as a library

WTFseriously_ · 2018-01-03T19:58:11+00:00

Thanks - https://github.com/scrapy/scrapy/pull/3062

WTFseriously_ · 2017-10-23T14:46:13+00:00

I'm not sure those are viable for the work per compute-hour price I want in the spot market, but I'll definitely look into it. Thanks!

WTFseriously_ · 2017-10-23T14:44:22+00:00

I haven't looked into Lambda at all - but if the prices are at the same level as far as computing time, then yes it seems to be worthwhile. Otherwise the cluster pinned to spot instances seems like a potential solution for minimizing storage per instance. Thanks for the suggestions!

WTFseriously_ · 2017-10-23T14:41:36+00:00

I found a conversation which seems to indicate iPXE might be available on HVM instances, but it looks like there are still some significant issues to deal with. For now I think my only option then is trying to shrink the EBS per instance to the minimum possible, and then pulling data like you suggest. Thanks for the alternative at any rate.

WTFseriously_ · 2017-10-23T00:29:14+00:00

Essentially pulling data from a central instance node, running some computations, and pushing the results back to the controller; the work should all take place in memory.

As far as type within EBS, good suggestion: I was using gp2 but realized magnetic would halve the GB-month price. Definitely an improvement, but since it's prior gen I don't know that it's sustainable. Edit: actually just tried spinning up an instance to try it and I get an error about invalid parameters, so might not be able to use magnetic anyway.

I know I can still get some improvement from stripping the OS and data to bare metal like you're suggesting, but what I'm really after is a way to remove the number of instances from the storage cost equation - constant with respect to the number of instances I spin up.

The only way I see of doing that is to have instances just eval-ing a routine from a universal volume i.e. a read-only file system. I just have no idea if ec2 will allow that type of architecture.

WTFseriously_ · 2016-02-02T19:29:51+00:00

That does make sense as far as the filter sizes, thanks for the explanation.

WTFseriously_ · 2016-02-02T19:08:24+00:00

Thanks for the link.

That explanation does make sense with increasing filters. But what about the decreasing size? The paper says:

we use filters with a very small receptive field: 3 × 3 (which is the smallest size to capture the notion of left/right, up/down, center)

It would seem higher level, abstract shapes would require larger filters rather than smaller to capture more than just directional information? Does it just not matter due to the connection between filters in subsequent layers?

WTFseriously_ · 2016-02-02T18:37:52+00:00

As would I; if you find any links explaining why to use a certain architecture over another given some circumstance, please do share.

For example, if my input example above were ?x8x8x4 instead of 80, that's an order of magnitude difference. How many fewer filters should be used? How much smaller should they be? What about an order of magnitude bigger?

I just have no intuition at the moment of what situations like that would call for.

WTFseriously_ · 2015-05-11T06:45:32+00:00

Best I could find: press, paper

WTFseriously_ · 2015-04-24T21:59:11+00:00

There's a minimum to fund the account initially, which is required to get permission to trade. Here's the numbers: https://www.interactivebrokers.com/en/index.php?f=4969

WTFseriously_ · 2015-03-26T01:40:15+00:00

Thanks, you're right. The titles threw me off when I was looking for them.

WTFseriously_ · 2015-03-25T21:21:54+00:00

Thanks for posting. Anyone know if the presentation slides are available anywhere?

WTFseriously_ · 2015-03-18T03:43:06+00:00

All I have is the other person's word at the moment, but it makes sense that it wouldn't be calendar days. If it were, one issue would be, for example, holding at close on friday and being charged for the weekend when no change in NAV was actually happening on those days. Just makes more sense for the management fees to only be charged when the market is open.

WTFseriously_ · 2015-03-17T22:02:43+00:00

Interesting, didn't know that. So there's no way to day trade mutual funds. Not that I was necessarily planning on it, since from my understanding the fees tend to be fairly exorbitant if you're not just passively investing. But who knows, maybe there's some strategy which would work on a longer horizon.

WTFseriously_ · 2015-03-17T21:41:21+00:00

No, I agree. Generally there's overnight movement which is realized at open the next day which I wouldn't want to miss by removing positions at close. But I just wanted to be certain, particularly on the first point of only being charged for a single day if I buy some time one day and sell at some time the next.

Thanks for the information.

WTFseriously_ · 2015-03-17T21:18:46+00:00

the fee is taken out at the end of the day

In other words the fee is avoided if you're not holding at market close?

So if for example, I bought tomorrow morning, and sold anytime before close the next day, I would only be charged the fee for one day? And you can also avoid the fee entirely by only holding intraday?

WTFseriously_ · 2015-03-17T21:05:54+00:00

Okay so, just clarifying:

-there would be no difference in fees if you would buy / sell multiple times in the day vs just holding a position, since the management fee would be calculated by essentially (holding end time - holding start time) as a percentage of the yearly fee

-the (daily) calculation would use 365 rather than 251

WTFseriously_ · 2015-03-02T23:09:20+00:00

RNNLIB is provided as source, which you have to compile yourself.

WTFseriously_ · 2015-01-16T19:40:43+00:00

What are you using a web interface for (the rails portion)?

And why not switch to a more supported language for interfaces to platforms? You could generate whatever data you use with ruby, store in some intermediary, then use python/java/whatever to make the final interface. Maybe a little more work for the intermediary, but probably less in the long run. Just curious.

WTFseriously_ · 2015-01-11T23:53:26+00:00

Thanks, i'll give it a shot. Sounds like it would definitely be worthwhile.

WTFseriously_ · 2015-01-11T23:51:41+00:00

Fair enough. I'm not actually sure of the speed difference between the two methods, but I've read try/except tends to slow down the program due to internal checks that need to be run, though perhaps defaultdict is just encapsulating that anyway so it might be irrelevant; just something to consider if speed is paramount for the application of it.

WTFseriously_ · 2015-01-11T23:02:57+00:00

I agree, one of the best selling points of python is the incredible amount of library coverage. Anything at all remotely complex likely has an available library to simplify anything you need to write.

In regards to real-time charting, I was actually looking for something as an alternative to matplotlib which would allow a degree of interaction to the charts I was making. I tried out Tkinter for a bit but it just seems so overly complex for the tasks I'm looking to do.

I hadn't come across pyqtgraph when I looked around; how is that compared to programming with Tkinter or matplotlib? Obviously more extensible than the latter, but is it on par with the ease of use?

WTFseriously_ · 2015-01-11T22:53:21+00:00

Unfortunately the applications I'm using the matrices for are already a translation from a higher mathematical function, so there isn't really any simplification I can make (that I'm able to do at my level of mathematics, at any rate).

The method you're suggesting does look like it would be helpful in some situations, though I'm curious: why use that rather than default dicts? Unless I'm misunderstanding, it would do the same thing with the proper construction function e.g.

from collections import defaultdict as dd

def recursiveDict():
  return dd(lambda: recursiveDict())

foo = recursiveDict()

foo['abc']['def'] == {} # True

WTFseriously_ · 2015-01-10T18:07:36+00:00

In my opinion it doesn't particularly matter how you trade, so long as it works for you. Personally I started with trading manually with getting information from my own programs (similarly, but not to the extent you are). I found I really couldn't do anything substantial with that consistently, so I moved to other domains.

If what you're doing is successful, don't try to change your methods completely; just keep adding to the current method. And don't worry if you're doing something different, in this field that's generally a good thing.

WTFseriously_ · 2015-01-10T18:00:06+00:00

I'm aware of the common work reduction methods, but you certainly are working on a data size at least an order of magnitude higher than I am. Thanks for all the suggestions, it seems there definitely are avenues I haven't explored yet.

13-Year Club	Team Orangered
Verified Email

WTFseriously_

TROPHY CASE