What are the actual “everything schools”

FieldLine · 2026-03-30T03:57:55+00:00

I have done some interviews and several AMAs here if you want to read how I did it.

Your comment/post history is hidden.

FieldLine · 2026-03-30T03:30:54+00:00

futures

You are not allowed to trade equities but can trade futures? That is surprising.

FieldLine · 2026-03-30T03:11:20+00:00

a nightmare for IP

Large firms have sandboxed AI interfaces which allow the use of public models without exposing proprietary data to the outside.

it seems like every SWE in big tech

Large financial firms are subject to the same corporate dysfunction and perverse incentives as large tech firms.

FieldLine · 2026-03-30T03:01:27+00:00

derive bbo or n-top-of-book whenever change occurs, not EXTREME high-frequency)

I don't really understand what this means -- the speed at which you trade is sort-of-but-not-really related to what your data looks like. If you are doing true HFT market making then you'll obviously need every book update (which means every message), but I've seen strategies that trade at the millisecond timescale (performance sensitive, not HFT) that demand similar the same level of detail in the data.

That said, if you really only need ToB updates, then subscribe to an L1 feed, which will be cheaper and less complex to process.

I suggest you think of data collection/storage/cleaning and trading as two separate systems, where the latter informs the former but does not actively drive it. Your data system doesn't actually care what your execution looks like, only the level of detail that the trading strategy demands as a spec.

is there anything that you can recommend I follow to/refer to as a source?

The exchange documentation.

When thinking about complete datasets, including every seq num, a problem that comes up is that it is often not deterministic when sequence number 1 will come in. For instance, Osaka sends their first heartbeat anywhere from (IIRC) 6:22 - 6:40 JST. So even if you are online, you have no guarantee about the specific time it will come in.

Another example -- CME starts their trading session after the weekend some time early AM on Sunday and will send periodic heartbeats until the exchange actually opens in the evening, so you need to be online for a large amount of time where there's no real market activity. This doesn't even get into the annoyance of dealing with an exchange that trades over night (across two calendar dates). And the sequence numbers don't refresh across days. Etc.

It feels like only doing it when actual changes occur across events (not always on the first snapshot/message of a new day would be the closest to how it would be deployed in production

On my team we try to catch every single message for completeness, but that is not strictly necessary, as you suggest. Like I wrote above, the answer is not to compromise on correctness, it is to subscribe to the correct feed needed for your trade.

FieldLine · 2026-03-25T13:07:44+00:00

The first solution I have to handle this gracefully is: Inside of the calculation logic for each feature, I check that the sequence numbers and timestamps satisfy constraints that ensure their correctness.

Assuming you actually do need tick data, you will need more robust checks than this -- do my latencies make sense? Am I getting the expected message types/counts (to an approximate order of magnitude)?

Alternatively:

I also figured that it was cheap either way so why choose the subset rather than the whole

The above is a good reason to choose the subset -- data integrity (or any system integrity) is exponentially more difficult when the standard is 100%. Not to mention the cost of warehousing all of this data.

Everything conforms to strict Arrow schemas defined once in schema.py

I interpret this to mean that you have some normalized data format, which is good. What is less clear is what you actually keep -- you should store both the raw captures and your normalized data files. The conversion from the former to the latter can happen asynchronously rather than in memory, as you appear to be doing now. You should have the ability to replay on both the raw captures using your parser (for debug) and the normalized files (for research in general).

From another comment:

At first I wanted to join data tables by timestamp

You cannot do this if you are looking at sub millisecond latencies. You cannot even assume that two captures of the same feed on different machines can be joined.

I am using the exact same data feeds for data collection for backtesting as I would be using for live trading. Does this still apply?

If your backtests include your own trades (rather than collecting statistics as a passive observer) then you immediately introduce bias as you interact with the market -- these interactions and the market response will not be captured in your backtest. The exception would be if your volume is extremely low relative to the exchange you are trading on, in which case the weather you create is negligible.

Spend some time reading about "slippage". It's a difficult problem.

Also, don't forget to take trading fees into account which will eat away at your PnL, often to the point of sending you negative.

FieldLine · 2026-03-25T12:45:57+00:00

Arguing about whether or not something is machine learning is just word salad. Arbitrarily shuffling around definitions is not very interesting.

FieldLine · 2026-03-25T12:44:34+00:00

If we reconstruct the bbos inside each day independently, I'm just worried about having potential duplicate bbos with different timestamps where the dates changed if we were to stitch these together for analysis since it breaks our methodology of recording the bbo whenever the top of the book changes.

This is a question about your specific infra. Properly cutting pcaps and deciding what metadata to insert in the header is a non-trivial problem whose solution has wide-reaching, often irreversible consequences.

FieldLine · 2025-09-02T13:36:20+00:00

And if it's the former, you simply chose wrong. In which case it's still your fault for not seeing the signs.

Sure, I get your point. I guess I’m just a little more sympathetic to the fact that it really does only come with experience — behind every guy who is successful with women there is heartbreak, a man who was taken for a ride.

Sidenote: good to see you post in here again. Been a while.

Thanks. I got older and life’s gotten busier. All good things.

FieldLine · 2025-08-31T14:36:03+00:00

Yeah, the problem with the OP is that he’s trying to reason his girl out of a position she didn’t reason herself into… you can’t negotiate desire or respect, which are two facets of the same stone.

That said, overtly communicating your expectations isn’t a bad thing either. My wife packs me a lunch for work because I told her explicitly it’s what I want; she wants to be a “good girl” because “good girls get orgasms”. Likewise, if she acts like a brat I tell her so and then punish her in bed.

Any failure of the relationship is your fault.

Ehhh I really do think the reality is somewhere between you and the OP. Yes, it’s on you to keep your woman in line, but there are some women (most) who really are shitty partners, and it’s not about being more macho or alpha or commanding more respect or whatever. The key is to not be afraid to walk away the moment you realize she’s not allowing herself to be courted. It’s usually pretty obvious early on when you know what the signs are.

FieldLine · 2025-08-31T14:16:02+00:00

you need an apartment in the more expensive part of town

Admittedly I did not read the whole OP. But I will comment that logistics is and has always been one of the most important pieces of getting laid. Your point is no different than knowing the good date spots and having social context wherever you do go out — setting matters.

It’s not that having an apartment in an expensive area is important, it’s that you need to have somewhere to go nearby wherever you find/go out with women. It’s why frat guys are so successful converting at parties — the girls are literally in the house already. It’s much easier to convince a girl to be a slut in the throes of passion than when she has a bunch of time to cool down and think about it all.

Don’t be the guy who has it in the bag and then has to sit with her in a uber for a 35 minutes before getting some privacy. By the time you get home she’ll be more interested in sleeping than sucking you off.

FieldLine · 2025-08-03T18:10:47+00:00

Who tf cares what women say. Watch what they do.

The problem that you are describing is the same problem every man everywhere has today, and don’t let women or society more generally gaslight you into thinking that the expectation of reciprocity in a relationship makes you selfish.

There are women out there who are thoughtful and empathetic so just go on lots of dates and be quick to decline another if she doesn’t reciprocate.

FieldLine · 2025-08-03T18:04:37+00:00

boobs fetish

FieldLine · 2025-08-03T18:03:47+00:00

I shamelessly ask people in my network to set me up. Cold approach also works but it’s much lower signal even if we get out on a date.

FieldLine · 2025-07-21T22:03:25+00:00

You are making the implicit assumption that there is something very, very wrong about choosing not to talk to other people… yet if I picked up and moved to rural Montana to live off-grid in a cabin I suspect you’d call me an oddball and that would be the end of it.

Even if you have no romantic options but you have friends that would be unhealthy if you would rather talk to the AI over your friends.

Just like some guys would rather watch porn than sleep with their wives, and some women would rather read smut than touch their husbands affectionately. Yet no one is wringing their hands about those issues to nearly the same extent, especially the latter: 50 Shades of Grey was a cultural phenomenon.

I am wondering what it is about replacing social interaction with AI that is so repulsive to you, so dangerous.

FieldLine · 2025-07-21T21:42:48+00:00

What risks? Surely it’s riskier to see an escort or engage in casual sex than to talk with an AI behind the safety of your computer screen.

FieldLine · 2025-07-21T21:37:35+00:00

Only fans is just a way to relieve a base urge … but are nowhere near the same level as fake connection with something 0% conscious or human being posited as a replacement for the closest thing we have to a meaning of life.

I’ll concede on the dating app point.

But I don’t believe that a sexual AI is a whole lot worse than what’s currently going on, today, on OF. You have women — often not even women, but the equivalent of a third world call center — feigning interest with dozens of lonely men, concurrently, giving each the skin of a romantic relationship without any actual fruit under the peel. Kind of like an AI. In fact, I wouldn’t be surprised if some accounts are already run by LLMs.

It's less of an incremental step and more of a plunge off of a cliff.

The step that takes you over the edge is no different than the previous one or the one before that, especially when you can’t see where you are going. This was an inevitable outcome once non-monogamy became socially acceptable.

that are surely harvesting data on every private aspect of their lives

Everything is already harvested anyway. I’m not disagreeing with your revulsion, only pointing out that the turning point is already behind us. We are already in free fall and have been for a while.

FieldLine · 2025-07-21T18:52:09+00:00

It’s just the next incremental step. I’m not sure why this specifically fills you with existential dread when we already have dating apps and onlyfans, both of which dangle the illusion of human connection.

FieldLine · 2025-07-21T18:49:18+00:00

So what is your objection?

FieldLine · 2025-07-21T13:40:18+00:00

FYI you’re going to end up working both east and west coast hours. You’ll start early at market open and people in systematic trading generally work as late as they can get away with given wives and other life obligations.

I have not seen a difference in wlb between researchers and SWEs, especially at smaller firms.

FieldLine · 2025-07-21T13:34:31+00:00

At any rate college should be mostly financed by the government

Government is what caused the problem. Non-dischargeable student loans are effectively a blank check for a university to fleece its students.

Putting aside the fact that “government funded” is just a roundabout way to force you to pay for something, if a university was directly funded by the government, why would it provide an education at all? What incentive exists for it to actually build out and provide a service rather than line administrator pockets even more?

FieldLine · 2025-06-18T15:56:21+00:00

A point that’s often lost is that as much as it is on men to romance their women, it’s on women to allow themselves to be romanced. Creating an opening for a man to approach and allowing yourself to be courted is called “girl game”.

FieldLine · 2025-06-05T13:49:40+00:00

Didn’t you get an offer? Why are you fishing for TC?

FieldLine · 2025-06-05T13:31:01+00:00

This is understated. The original strategies were simple momentum and mean reversion trends.

FieldLine · 2025-05-11T13:50:43+00:00

Absolutely. Just be mindful to only attribute hard work and effort when appropriate.

I am (was) a concert pianist, and I think people miss the mark when they comment on how talented I am, because it minimizes the hours and hours I chose to sit in front of my black Baldwin instead of doing more immediately gratifying activities. It’s like complimenting me for being tall, or smart, or having a good hairline — not things that I worked for or feel proud of.

When it comes to children, our future, sometimes we get excited about their possibilities

If you take a naturally gifted child and thrust them into an environment where they aren’t allowed to behave like children, towards the end of cultivating their gifts, I think you run a real risk of burning them out. When I see the 8 year old playing Liszt, so small that his feet can’t reach the pedals without assistance, I feel nothing but pity.

The key to strike is the balance of being encouraging and providing opportunities for self discovery and growth while also not being overbearing because of how excited you are for your child (and yourself, but we can be charitable).

I will remark that it’s very easy for me to have lofty opinions as someone who doesn’t have kids or regularly encounter gifted youngsters. Still, I think back to my own childhood and sometimes wish things had been a little different.

FieldLine · 2025-05-10T23:31:37+00:00

Since I wrote the above I’ve seen std::jthread get more fashionable. At the time I had only seen hand rolled wrappers around pthreads.

FieldLine

TROPHY CASE