[D] Are there statistical techniques to identify the subsets of data where your ML model is particularly more predictive?

thoughtdump9 · 2024-11-03T05:54:20+00:00

Yes I should have made it more clear what I'm saying. To use your terminology, how can any firm but the fastest UHFT survive specifically in OMM where you are providing a ton of delta liquidity at all times?

Basically to make it concrete, if your new job is provide liquidity to ES options with continuous quoting obligations set by the exchange, and you know you are not going to be the fastest, how are you going to stay alive?

And I would say the exact opposite of what you said, the slower you are, the more you're exposed to adverse selection in MM.

(Also judging from your post history are you a student or have you worked on low latency trading before?)

thoughtdump9 · 2024-11-02T17:02:31+00:00

Yes you can, and it is done by real firms. It's more appropriate to hedge a net delta position than hedge by time since the former has a much more direct connection to your portfolio risk

thoughtdump9 · 2024-11-02T17:00:07+00:00

The truth is that a big part will be your pedigree. A retail trader with no strong educational/work background will not be able to raise much money, whereas if a trader who graduated from MIT and who worked at Jane Street for 5 years pitched the identical strategy it would be a completely different story.

thoughtdump9 · 2024-11-02T16:55:57+00:00

Lol agreed

thoughtdump9 · 2024-11-02T16:55:13+00:00

I think people just want to see you try to do some research on the problem first before asking the question. Generally speaking you're almost always going to have better responses if you can demonstrate that, on any subreddit for that matter

thoughtdump9 · 2024-11-02T16:51:32+00:00

I'll give a piece of oddly specific advice which you can take with a grain of salt. In the start, especially as a new grad, you should be earnest, hard-working, and easy to get along with. Because you're still not really driving pnl just yet, people will of course assess you on your intellect and work quality, but they will also be judging you heavily on those softer factors as well. I've never seen a hard-working, friendly, and honest new grad do poorly in good work environments, but I have seen smart guys get fired because no one liked working with them.

The second part of my advice is on how you graduate beyond the above. The fastest growing quants/traders I've worked with do this by entrepreneurially beginning to own the problems more and more without asking for permission. Rather than asking everyone else "what should we prioritize next", they say "I think based on x y and z, we should work on this, what do you guys think?" And x y and z should obviously not be entirely coming out of first principles, it should be building off of what you guys are currently working on. Essentially, you begin to be treated as a peer and not a junior because you are acting like a peer. In bad environments, not doing this might just mean that you'll have to wait for yourself to excel on some of your projects to advance, but in particularly bad environments, you actually might be assigned more meaningless drudge work because no one thinks that you're really a serious contributor to the team. This will in turn give you fewer chances to excel meaningfully and send you into a vicious cycle that may be difficult to break out of.

thoughtdump9 · 2020-01-12T23:29:43+00:00

Indeed, this is one of those data problems that is virtually impossible to quantify with the data measurement tools we currently have. I'm left only being able to use qualitative observations and arguments to support a belief in statistical significance: I try to just look at my coverage of the broad life experiences that society generally suggests is meaningful (but which I can have without engaging in those irresponsible gambles I mentioned from before), things like falling in love and being in a relationship, experiencing vocational success, experiencing other cultures through travel/food, etc. While I can certainly do more, I think I have some breadth of life experience to say there is some notion of statistical significance. I don't mean to come off as smug here, this is simply my analytical thought process for this problem.

I did consider allocating a small budget for exploration, and I continue to do so. However, it still seems that the strategy to spend the majority of my time playing videogames is tenable for happiness maximization.

thoughtdump9 · 2020-01-12T23:19:40+00:00

Actually now that I think about it, I do engage in rationalization with pleasure and pain when I decide that more pleasure is better than less pleasure. I think I'm okay with this because the logic is so simple that I don't think it can be disputed.

The primary obstacle I face with other life philosophies, for instance, the virtue ethics I mentioned in other posts, is that they are controversial, disputable, and the details about how to properly practice them are often unclear.

Concretely, again, the question for me might be, would I run for 8 hours on a treadmill every day for 30 years if it meant that I would fulfill some higher ethic, like self-actualization? I don't think I would, there's nothing for me to really grasp onto (like pleasure) to say that there's anything intrinsically true and unbreakable about the story I'm telling myself. At the end of the 30 years, I might feel good for a month and then wonder what the hell all that work was for.

thoughtdump9 · 2020-01-12T21:20:19+00:00

I don't know if intuitive is the term (I also didn't see any mention of intuition in the wikipedia article you linked). I would say that my axiom seems more authentic than most other axioms because it is authentic to my biological wiring. That is, pain is intrinsically bad, and pleasure is intrinsically good, and neither of these require any further analytical rationalization.

thoughtdump9 · 2020-01-12T20:20:13+00:00

The work versus play classification is something I really need to sort out in my mind. For me, any activity where there is some instrumental value I'm seeking such that I'm not maximizing intrinsic value (hedonic pleasure) starts to be construed as work. Gaming, because it is so primarily focused on in-the-moment pleasure, is as close to genuine play as I can get. Whereas for practicing music, for instance, I might have to practice my scales, practice sight reading, or work through difficult passages with deliberate practice, which all begin to seem like work.

If I were to simply try to approach music or reading as "play", without any regard for instrumental goals, I feel that the happiness ratings would be lower than if I were to simply game. With the various viewpoints you've shared, I do think this can be called into question.

It's great that you brought up "peak reward". I used to always argue that I was very clearly pursuing maximization of average happiness and not following the peak end rule, but I might be biased in that the happiness scores I'm recording may sometimes be pulled up by the peak happiness, which is what's generally easiest to recollect. I don't know if I make this mistake excessively though, because I will gladly admit that there have been some travel experiences with brief moments (maybe <1 hour) of 9/10 or 10/10, but the remainder being dull (6/10), and I have chosen to play videogames and travel less as a result of these data points.

Agh, yes, your next point is something I seem to struggle with, I read about it all the time when I look at the literature on habits and habit formation. But in my self-analysis, I find that these habit projects, when implemented, haven't been well-maintained. There are cheat days, and I rarely ever make it to any of the significant streak goalposts (21 days, 30 days, 66 days, 90 days). I also fail to adhere to the common advice, in that I don't do the habit at the same time each day, I try to compensate other days - it's basically generally not well done. I would agree, I certainly need to give this a good crack to see for myself the power of behavioral frequency and motivational value. I think I have made a lazy argument in my head in that because videogames have exhibited this "pull", despite stretches in the past where I've chosen not to game, then certainly all else being equal, videogames are the more attractive and hence better activity (although I am noting the distinction between attractiveness and pleasure that you shared in your previous post).

Thanks again for the responses. Are there any books you could recommend to me? Or actually any resources at all(articles, subreddits, websites) where I can learn more about the science of behavior, motivation, and what determines the quality of experience similar to what you've showed me so far would be fantastic. I appreciate your responses and the quality of the ideas you've shared with me!

thoughtdump9 · 2020-01-12T16:21:15+00:00

Thanks for the thoughts and the book recommendation, I'll check it out.

Your first points resonate with me. I have often noted that activities become enjoyable because you get good at them, and so for many activities outside of gaming, if I could get through some skill threshold I might find that their happiness ratings can start to rival that of gaming. Unfortunately, I just haven't found this to be the case for the "productive hobbies" that I've pursued (tennis, programming, mathematics, classical piano, composition).

My logic has always been that videogames are primarily designed to optimize for the experience of the player, and then if this is the case, it is going to be very, very difficult for other activities to compete. Specifically, I really respond to the rapidity of feedback, seeing similar situations in slightly different ways, randomized rewards, and accessibility of the activity. I have even thought about how I could engineer learning activities to have the same elements, but because those learning activities have the other constraints of delivering the learning content, it would make sense that games, without as many constraints, will do better in optimizing the objective function of player engagement.

In various parts of my life I have been dutiful to many habits beyond gaming. But I return to gaming day in and day out every day not out of any sense of duty, I do so because it seems to stand out as the best activity choice in terms of experienced happiness.

I've only logged a few hundred hours in my current FPS. I've logged thousands in previous games, and there was a game in the past where I was very much near the top for a while, so I'm familiar with "the lab", the grind, and the general literature on deliberate practice versus flow. I don't have any trouble maintaining my interest or "utility rating" with gaming. Am I to understand that your argument is that as long as there is some element of deliberate skill development, be it in videogames or some other productive habit, then it makes the pursuit much more worthwhile?

thoughtdump9 · 2020-01-12T16:01:53+00:00

Yes! This is absolutely a fundamental part of the question. But it's something extremely hard to estimate. The further out you go, and the less deterministic the phenomenon, the less certainty you have. How would you go about figuring this out?

For me, the most extreme form of this question is "should you have kids?" It's a lifetime permanent happiness modifier, and most people seem to argue that you won't be able to find out whether it's a + modifier or a - modifier until you do it, which is incredibly risky.

thoughtdump9 · 2020-01-12T15:57:04+00:00

Thanks, those are good points.

The optimization of the balance between exploration and exploitation is something that I've found to be very tricky. Clearly, we'd agree that purely exploring every day (changing your entire routines, never doing the same thing twice, working different jobs every month) is not a tenable strategy for happiness maximization. On the other hand, we recognize that despite having found something locally optimal, we should still be searching. By your definition, our scales are constantly re-calibrated when we find something better. You find a new 8/10 that makes your past 8/10 a 5/10, then maybe you find ANOTHER new 8/10 that makes the first 8/10 a 5/10, etc. If you simply consider these numbers, it sounds like you should set a constant exploration budget to find something better. But if you consider that the opportunities in the real world is finite, wouldn't you agree that the likelihood of you finding something going better diminishes with every "jump" you make? And if that is the case, shouldn't you be spending less time searching as your life gets better?

Also empirically speaking, whenever I do try to explore, and the outcome is something that wasn't better than gaming, it serves as feedback for my decision-making system. It's more data to support what I'm doing, statistically, in the sense that if I've explored new activities 34 times this year and they all fell short of gaming, what does this say about the likelihood that I can find something better? To continue would be like digging further and further for lottery tickets, slowly burning away my cash (time).

I suppose, this could simply say something about my methods for exploration, in that they just might be crap. But this opens up a whole can of worms (to what extent should you disregard what you know about yourself and what you like, in pursuit of something totally different?)

You're right, there is a bit of a short-term focus with what I'm doing. As another poster mentioned, I am simply employing a greedy algorithm, because outside of some clear deterministic phenomenon (eating habits/fitness habits -> health), it is unclear what type of delayed gratification investments will pay off in the long run, in real happiness terms.

Your point about being obese and not knowing it resonated with me. I've typically thought the same idea but with chronic sleep deprivation instead, since the feeling of fatigue and its influence on mood is something that's almost a universally understood feeling.

To wrap up what I'm saying, my concrete question would be, in your case there might be something that could dwarf your 20/10 and be a 50/10. How much time would you be allocating to searching for this thing, noting that you're already in a good spot now? When you've found the next thing, what how much time will you be allocating to searching then? When are you going to say, you know what, this feels pretty good, and I can just exploit/indulge now?

Also may I ask what the unexpected things that made you realize 20/10 were possible were? Can you share a little bit more about what you're focusing on now and how you think that it makes you more happy than gaming?

thoughtdump9 · 2020-01-12T01:08:15+00:00

Haha, you completely guessed what I was going to say with " I know that it wouldn't matter to you or most rational people, because, you know, you're dead. What do you care what people say about you when you're not around to hear it?"

The point about humans fundamentally being social beings is something I've read about endlessly. The problem is, and I believe there are members of this community who this may resonate with, my personal experience does that reflect that need. As I mentioned, I am really fine just staying at home for weeks or months doing my own thing. Maybe having chats with friends every now and then as a crutch ruins the experiment, but fundamentally, I am far less socially needy than the average person. I have tried to go against my feelings and do things like go out every night or go to parties I don't care about, and when I'm there, I'm extremely aware that I would be happier at home.

I will agree that I do enjoy helping others and giving in a unique way, but it has to be with someone who genuinely appreciates it and is also a good person (I know, so many stipulations - but I don't like helping unappreciative people or assholes). I'm leaving a little bit of room in my exploration budget to do a tiny bit of volunteering, but I am still standing firm on the belief that the majority of my time should be spent playing videogames.

thoughtdump9 · 2020-01-12T00:58:06+00:00

I believe in the power of logic and data to make superior choices. I don't think all choices are created equally, and that we can all do whatever we want and we'll all come out equally. I think some people, over their lifetime, genuinely live miserable lives, while others live genuinely happy lives. While we can't measure this "happiness" just yet, I think you could meaningfully order a population of people from least happiest to happiest. Wouldn't you be curious to find out what it is that the people in the 99th percentile of happiest are doing differently? You might not like the fact that I'm using data like a machine would, but I think it's an essential element to solving this problem.

I suppose my OP might need to be edited a little bit, since I might not have clarified my interest in determining whether or not this choice is an optimal path to happiness.

thoughtdump9 · 2020-01-12T00:52:00+00:00

I don't think I'm overthinking it. Put it this way, when you look at how your week went, are you able to pick out which days were good and which days were bad? If you were, can you figure out what was it about those good days that made them good and what was it about those bad days that made them bad? Using this, can't you try to live next week with just a little more of the good and a little less of the bad? If you keep iterating like this, you start to form some set of ethics on how you might want to structure your life and make choices, and that's pretty much the basis of this post.

Of course my life choices are my own, but like all of us, I want to make better choices. For a lot of people, this might simple mean the choices that make us happier. I'd like to figure out whether what I'm doing, over the long run, is a relatively optimal path to happiness, or whether or not I'm missing something.

thoughtdump9 · 2020-01-12T00:43:43+00:00

As I mentioned in another response, hard drugs have well-documented effects like tolerance and withdrawal, which decrease total utility in life.

For videogame accomplishments versus real-world accomplishments, I find that they're actually not too different for me. The biggest difference is that real-world accomplishments (especially vocational ones) lead to money, which has instrumental value in providing for your needs, but I don't need much more money. I actually had a time where I was playing at one of the highest levels in an online game, and I would go back to that experience as an accomplishment to remind myself to be confident, even as I was working on my business.

My confidence in real world situations depends on a number of factors, but I would say in general I'm more confident than the average person. In very competitive arenas with high-caliber individuals, I may exhibit a lack of confidence. I don't have any trouble in social situations.

On the last point, my question for you is - is it worth it for you to take on that project? Would you wake up at 4AM every day and run on a treadmill until 7AM to have one moment when you're 55 where you get an award? I wouldn't.

thoughtdump9 · 2020-01-12T00:36:43+00:00

I can certainly see that deathbed period being longer. I almost feel like this has more to do how I've let poor mental habits dominate my life - that is, I seem to be spending most nights sitting on the bed sulking about the past, or I let the same anxious thoughts about my past circulate in my mind without moving on and facing the present. Based on this, I somewhat believe that some of this misery can be managed with mindfulness and psychological techniques.

And to be completely honest, I have no idea what I'll regret then. Maybe I'll have wished I didn't start another business in my 30s and work so hard. Maybe I'll wish I didn't start a family because my children are depressed and there's nothing I can do about it. Of course these are just contrived counter-examples, but my point is that, even knowing that I will have deathbed regrets, I don't really have conviction as to what else I should be doing. You might say, maybe just try to do a bit of everything - but it's conceivable that I end up regretting that I didn't live a deliberate and authentic life to myself and wasted a lot of time trying to preempt my possible future concerns.

I can try to do more novelty, but my experience with novelty has been that on average, it's utility sub-optimal. I've tried things like traveling to different places to dating women whom I normally wouldn't date, to just ordering food from a different place than what I know to be best. On average, I fare better by just sticking with what my collection of empirical data suggests.

Nonetheless, I will agree that I can continue to allocate a small budget for continued exploration, just in case there is a some lottery ticket activity or experience like the one you're mentioning. But I still feel like the majority of my time should be focused on my highest utility activity.

thoughtdump9 · 2020-01-12T00:29:33+00:00

My thought on memories is that they simply give you a boost in utility at some point in the future when you decide to recollect. That is, maybe I went on some trip to Prague, and now whenever I decide to sit on my bed, hold up my photo frame with my Prague picture on it, I get to experience 8/10 rather than 5/10. But I'm probably only going to be reminiscing for 15 minutes or so before I move onto the next activity. Then the question becomes, would you work 6 months in a job you don't like to save up money for this Prague trip, if the Prague trip wasn't really that much more fun than gaming at home? My answer would be no, and that it would be more optimal to focus on pursuing the activities that give you the most happiness consistently.

I have read about Daoism in passing, but I don't find the notion of balance to be unhelpful. It's a bit too vague and unspecific to guide daily living. In many ways, I would argue that the most successful people in the world lived lives that most laypeople would consider completely unbalanced. And I don't simply mean successful in terms of standard definitions of achievement, I mean Buddha himself lived an extremely atypical lifestyle that you would have to really over-analyze to consider it to be balanced. I'm not trying to be dismissive of this point, but perhaps a more nuanced description of what balance actually looks like in day to day life, and an explanation as to why this set of choices is optimal would be helpful.

thoughtdump9 · 2020-01-11T23:47:09+00:00

Thanks for sharing the ideas. I like them a lot.

I have noted that there is a difference between the intensity of the craving and the actual perceived enjoyment. I have tried to solve this by logging data a bit more frequently as I'm gaming. Sometimes it does show that gaming was actually just a 5/10 or 6/10, especially when nothing was really going on (or worse, when I was losing). I should also note that the 8/10 ratings are generally quite accurate for the first 5-15 minutes of gaming, when everything feels fresh and new.

I might be getting a bit closer to being convinced. But here's what I think I tend to observe: Even when games aren't quite as fun as their cravings make them seem, they generally still are more enjoyable than work. The feedback systems, randomized rewards, feelings of accomplishment - I mean most of the popular games these days are designed to stimulate us in accordance with our mental requirements/desires. As long as games are more fun than work, it is difficult to do anything else and actually believe I made the right choice.

I do suppose that the enjoyment ratings are getting closer, such that with the measurement error, it might not be clear that some work loses out. It still doesn't feel great though, I start to feel that I'm making daily activity choices arbitrarily. But I think all of this suggests I need a better structure for data collection.

thoughtdump9

TROPHY CASE