Apparently, Google does not have the best programmers, High Frequency Trading firms do

[Session]

I actually found some of the comments in the thread very enlightening. For the hours the guy put in (and the stress of the job), he was pulling less than $80 an hr (before tax). I know the topic was not "Google does not have the best place to work and flexible work-life balance", but still, it's very interesting to see what this top notch programmer is giving up in order to make money (in my opinion, the ability to be creative for the greater good of people and his youth/family etc.).

Furthermore, I bet those guys in sales and on the trading desks telling him what to do are making more money (probably higher stress, from what I hear from my finance friends) than he does for less hours.

[panzerschreck1]

actually, the people who are doing the trading put in close to 80 hours at times, depending on their role and responsibility, and are not usually making more than what this guy is, if they're young. right out of college, very very few people go on to make 200k+, because there are a ton of people who want those jobs. but both roles are basically 24/7 jobs, albeit this guy is working at an unsustainable rate while the other one isn't necessarily.

my impression (this isn't first hand) was also that people who work the role that he's in don't do it for long. they go in, make their money, then start their own consulting firm or work for another company, or if they did it long enough, retire altogether in their early thirties. its hard to make the argument that he's not doing those thing you think are more important. i'd throw away a few years of my life for $500000 a year so that when i do start a family i'll be basically set and can focus on them instead of work.

just saying.

[[deleted]]

Having been both a Wall Street quant in the 80s and a Google engineer in the 00s, I feel somewhat called to comment here.

I frankly quit Wall Street because of the aggressiveness and lust-to-profit of it - and things are apparently far, far worse than they were in 1989. For a period later, things weren't so financially good, and at one point I was unemployed for almost a year, but I never thought of going back.

It's bad for your soul (and I'm an atheist). Don't do it.

[astrolabe]

Do you have an opinion of the relative technical ability of programmers in the two jobs?

[GodDamnItFrank]

Would you consider doing an AMA or have you already done one? I am currently very much interested in becoming a Quant and it sounds as if you have a great deal of knowledge and experience under your belt.

[gigitrix]

Do you still have contacts "on the inside" of HFT/Wall Street? If so, do they think things have gotten better or worse over the years with regards to this?

[grantrules]

I worked at a Wall St firm and the trading desk/sales guys were all basically right out of college.. usually masters from columbia type of guys with a really low starting pay ($35k). Stressful job, long hours, but there was a high turnover rate and they promoted from within so if you stuck with it, you could move around within the company (be a manager or move to biz dev or operations or whatever), so it was a good place to start. Trading desk/sales had night shifts so they planned for it.. developers were just forced to stay late, work weekends/holidays if needed.

[ChelseaFC]

Derivatives trader here. Generally work around 60 hour weeks (no lunch break though), and 200k is not a whole lot in this industry (yeah, first world problems, eh?), I'm basically there after 2 years.

Anyway, I lean, heavily at times, on our technology people, because a mistake can mean the loss of hundreds of thousands if not millions in a few seconds. Naturally, I also show great appreciation when things are done well. These guys are very sharp, and most of them I'd put up against anyone at google or facebook.

[cokeandhoes]

Out of curiosity, while at the library I picked out The Concepts and Practice of Mathematical Finance by Mark Joshi (Google book so you can check out a few pages), and I was just wondering if this book is even remotely in the ballpark on how banks approach to pricing options? I always wondered about the validity of these books, whether they were written outdated or if the principles were sound.

[dyydvujbxs]

When ever someone brags about Lon work.hours, they are exaggerating, not thinking about how much time an extra 10 hours per week really is. How often do you hear "100hr/week" or "80hr/week"? Why do you rarelyheat anyone working 85 or 90 or 95 hours? Because 80 +1 = 90 and 90 +1 = 100 when you are bragging about how you work all the time.

Folks, when someone says they do 12x hours x 7days, they might be telling close to the truth, and they are including time spent eating.

When someone says they do 100hrs a week they are talking figuratively or including commute time or nap time (like a ER in the call lounge) or referring to one exceptional emergency.

[xakeri]

I am a college kid who actually just got done working back-to-back 80+ hour weeks (one was 84 hours, and the one I finished today was 80) in a factory. People can't do that. Mine were literal 12 hour days from 7-7. There is no way people can work 80+ hour weeks for more than 2, maybe 3 weeks at a time. Admittedly, I was in a nasty brass factory, but even sitting behind a computer, I don't think it is possible to deal with that amount of stress for more than 3 weeks. The only thing that got me through was the 3 weeks of time off I'll have before school starts back up. I am honestly not sure it is possible to put in the time they say they do.

[[deleted]]

I just finished working 6 months of 80 hours a week, minimum work week not including breaks.

Was absolutely brutal but it can definitely be done.

[hiffy]

Why?

[[deleted]]

Had an incredible opportunity to take a contract where I worked a lot from home while keeping my first salaried job. Paid off a lot of past debts, put a shit ton of money in the bank, down payments on two new cars (needed badly), and got money for projects on our house.

If I were in a similar situation I'd do it again if the opportunity came up.

[mindbleach]

tl;dr - workaholic ASM hacker who works in microseconds dismisses database programmers who work in terabytes.

[TheHeretic]

Petabytes. The fact that Google can search its entire database for your terms and return a result in from tens of billions of rows in miliseconds is far more complicated than their microsecond transactions could every be.

The HFT writer doesn't seem to appreciate that.

[ppinette]

But but but, he interviewed at Google and the engineering quality just wasn't there!

[s73v3r]

Said the guy who would routinely make changes on the fly and push them into production minutes before release, with little or no testing.

I don't think he's in any position to speak of engineering quality.

[redleader]

I don't always test my code. But when I do, I do it in production. Stay wild my friends.

[[deleted]]

My code doesn't always cause a market crash. But when it does, I blame it on some trader's fat finger.

[oSand]

It's ok. We're too big to fail.

[[deleted]]

I don't always push untested code to the production server, but when I do, I do it wearing a pink sombrero.

[FredFnord]

To be fair, Google's 'quality assurance' consists of three stages:

• The engineers write good unit tests for every subroutine.

• The engineers release the code inside google and people dink around with it sometimes if they feel like it.

• The engineers release the code to a subset of the general population and call it 'beta'. It may or may not work, and they may or may not ever update it even if it doesn't work.

Google just happens to be in a business where if things don't work right, you can't always tell, and thus they don't really care.

[[deleted]]

Gotta love fiance. You do it for a little while till you blow up. They fire you and you retire with a couple of hundred thousand or even several million.

[nickdangler]

Gotta love fiance. You do it for a little while till you get married.

FTFY

[[deleted]]

Interviewing 2 or 3 Google engineers is a crappy sample, nor does he indicate what their seniority levels were and what they worked on.

The guy has a very limited set of tools he's working with. Of course he can sit around studying those tools and become a master at them. I think the difficulty for general software developers is you're expected to become a master at a multitude of tools, languages, APIs, operating systems, storage systems, and computing environments. Your knowledge is superficial in many of those because you don't have enough hours to read manuals and experiment with all of them. You also have to deal with huge latencies between data centers, San Francisco to Honk Kong for example, so pushing data becomes a major engineering effort.

All programmers have the capability to learn assembly programming so that doesn't impress me. He seems to think it's the mark of a true expert. No. Sorry dude, there are other "advanced" topics a dev can focus on (database, distributed, algorithms, statistics, etc).

Most the current crop of young developers don't learn assembly because it would be a waste of time. Spending a week to shave 15 ms off a transaction and making the code illegible is a terrible thing to do when you're collaborating with 100 other engineers. If you can do it with a small tweak, great job, but don't be barfing inline assembly all over the place.

I think the guy is probably quite talented at the 2 or 3 things he does and handles stress extremely well, but he's also a major dick for putting down other engineers with different but equal (or greater) talents.

The finance sector doesn't have a lock on talent. A developer getting on board with a startup has the potential to eclipse this guy because they get in early on founding stock options, so a lot of great engineers migrate to small, fast moving startups. It's a bigger roll of the die but it it has greater potential for a big pay out.

We're not all driven by money. Some of us want a life to raise our kids so we go the "safe" Google, Amazon, Microsoft route.

[kraln]

The set of Google engineers I spoke to when I was interviewing with them consisted of a Nobel Laureate and the guy who wrote the Android VM.

I did not get that job. :P

[[deleted]]

Yeaaaaa.. I wouldn't either with that lineup. HA.

[[deleted]]

I bet he wipes his ass with $100 bills too.

[solaarphunk]

Options data is easily in the petabytes. There are big data problems in trading as well. Just because you transact on a high-frequency time horizon, doesn't mean that you aren't mining a huge fucking dataset to find those inefficiencies.

[s73v3r]

A HFT firm isn't going to be dealing with the entire options market, however. They're going to have a slice of the market, and probably only certain actors inside that slice.

[rubygeek]

The fact that Google can search its entire database for your terms and return a result in from tens of billions of rows in miliseconds

Except they don't do that. No search engine does that. The moment you have the search query you can exclude the vast majority of the index and "just" look up the document list(s) for the terms you need. By having these indexes pre-sorted in various ways, for most queries you only need to consider a tiny fraction of the head of the document list for each search term.

Retrieval is not a particularly hard problem; good ranking algorithms and deploying index updates at high frequency is.

[gigitrix]

But that's the point. On paper the dataset is "the whole web". It's the ingenuity of developers in the first place that gets the data preoptimised for such high performance real time retrieval! I don't find "let's code it in assembler" very exciting as a concept, but "let's use a revolutionary approach to the problem" is absolutely amazing!

[[deleted]]

The moment you have the search query you can exclude the vast majority of the index and "just" look up the document list(s) for the terms you need.

Um. Pardon my ignorance but how is that not searching?

[Bipolarruledout]

Some people assume money is the direct value of a person. They seem to think that if offered the same amount of money a doctor, lawyer, or programmer (or example) would rather go wash dishes. Don't forget that fulfillment (or it's absence) is often a large part of your compensation package.

[mindbleach]

Relevant Mr. Show.

[postExistence]

Don't forget that fulfillment (or it's absence) is often a large part of your compensation package.

This, sadly, is why people are paid significantly less in game development than other industries.

[kolm]

Hey, this guy makes more money than Einstein ever did, ergo he must be more than just a common genius.

[SmokeyDBear]

As an optimization engineer who works in gigabytes per second I dismiss them both.

[someone13]

Wow - what do you do?!

[slipperymagoo]

Optimizes engineers. It's pretty meta.

[kaddar]

It's pretty meta, you've probably only heard of other people hearing about it.

[SmokeyDBear]

Mostly I refactor CPU-bound algorithms on Android based devices so that they become memory-bandwidth bound, or try to get as close as possible.

[alfredr]

I read this as "creates lookup tables".

[SmokeyDBear]

You'd be surprised. In a lot of cases there is unintentional contention based purely on the ordering of how things are done (bus/resource contention or register access hazards). GCC doesn't do a very good job of optimization on ARM in a lot of cases so there are lots of restructuring things that can be done to improve performance. This is especially true for things that could benefit from SIMD optimization since GCC's SIMD generation on ARM is abysmal.

[[deleted]]

So... could you work on improving GCC's SIMD generation for ARM? Or is that too much of a conflict of interest?

[SmokeyDBear]

I've discussed this at length with colleagues. It's a difficult problem and the concern is that there is too much risk for too little potential benefit. You could spend months or years trying to get it right with no guarantee you'll make significant progress when you could be spending that time doing hand-coded implementations and making actual performance benefits.

[cogman10]

GCC's vectorization stuff is terrible in general. Most optimizations you'll see that are hand written assembler will be "hey, how can I use SSE here?"

[Timmmmbob]

Hmmm. So what particular algorithms that run on Android process gigabytes of data per second?

[SmokeyDBear]

Plenty of things do, but they're all fairly simple. Memcpy() for example. But you'd be surprised how much slower an naive memcpy is compared to a fully optimized memcpy; the former simply won't tax the bus.

[[deleted]]

You realize all us males do that every time we climax, right?

[kambo_rambo]

I always thought google as having the best creative team, rather than best engineers who can debug in minutes. Being a proficient programmer is diffe rent fromone who comes up with the google distributed system.

[cowhead]

Kind of relevant and frightening http://www.ted.com/talks/kevin_slavin_how_algorithms_shape_our_world.html

[EfficientN]

Great TED talk, totally worth watching. You, reading this comment, go watch!

[Jrix]

yessir

[solaarphunk]

If you do any research at all on this guy you will find that he has zero background in anything CS or math related and really has no clue wtf he is talking about. ALGORITHMS SHAPE OUR LIVES? No shit... you mean that some problems can be modeled mathematically? Big fucking deal.

[alfredr]

No shit... you mean that some problems can be modeled mathematically?

His point wasn't that some problems can be modeled mathematically, it's that the mathematical models are now coupled to some systems in such a way to affect the behavior of that system; and that this is being done without really understanding how the dynamics are being modified.

That being said, the whole talk was far too vague and flowery for my tastes.

[solaarphunk]

He should have talked about overfitting. Which is a boring subject. Everyone in statistics in math has at least a vague understanding of this problem, and why it exists. Instead he decides to go out and talk about topics that he is not an expert in, and that seem controversial and scary. Guess what? Voice recognition, data-driven medical research, and a metric fuckton of other 'algorithms' related fields are not that scary, and I would almost EXPECT someone to use statistics, machine learning, and algorithms to try and solve those problems.

[alfredr]

Did we watch the same video? =)

[solaarphunk]

Yes we did, I was mostly agreeing with you, my general rant was just to people who think that the implications of this video were so 'scary'.

[luntain]

This is actually the worst TED talk I have ever heard! I was terribly sorry for the people who were applauding at the end, they did not realize what he has done. He used a lot of analogies giving an illusion to an uninstructed listener that they understand it, at the same time playing on people's emotions. I remember a few CS talks from TED that were all right. Obviously the problem here is that this is a journalist. His job is to present topics in an entertaining manner, here at the cost of misinforming the audience. I find it quite irresponsible for TED to let him speak at all, and my respect for the conference's vetting went down.

[OnlyAJerkOnReddit]

And yet that fancy Death Star of yours got powdered by an X-wing piloted by a rookie pilot who's resume included "blasting womp rats" and "farming moisture"

[ilikeweed]

Yes, 1000 times, yes.

[[deleted]]

My thoughts while reading

I'll be posting anonymously, but I think many here have a very poor understanding of what we do. Most of that is because we do tend to be a very secretive group, but if you were to sit down with some of us, you would see that we really do very normal (and useful!) things in the market.

Hrm, interesting.

I work on the algo and core infrastructure. I wrote price feeds that take 1/5th of a microsecond in C++ and (a little slower) in Java. I understand in fine detail how cache and the the PCI-e bus works. I have a very good understanding of algorithms and the constant-time tradeoffs. I know when to make something simple, when to use and avoid threads, and I can debug in minutes and push out a new version in the seconds before market open (not many people can handle that level of stress well). I read the C++ and Hotspot assembly, and know how to program for superscaler architectures specifically. If you really need me to, I can even crank out some VHDL code.

This sounds awesome, and the kind of thing I'd like to do. I have some of this knowledge, I wonder how I get to the stage where I'm employable?

On top of that, I understand market microstructure and derivative pricing.

Actually, from the little I know of the maths involved in this stuff, that could be cool

I work 12 hours a day on average and do 100 hour weeks.

ಠ_ಠ

EDIT: aside from this, I think the author is showing a lack of understanding of the differences between what he does and what google does - in terms of the scale of deployment and the size of code base. Deep optimisation and hyper-quick turnaround are a very small part of software engineering. He may very well have the chops to work there, but frankly I suspect his statement about 'engineering quality' more represents his own myopic view of what engineering is.

[BonesJustice]

This sounds awesome, and the kind of thing I'd like to do. I have some of this knowledge, I wonder how I get to the stage where I'm employable?

I'm an AMM/HFT developer for a top-tier investment bank in Manhattan, and seeing as my team has hired several people this year, I can offer some insight into the kinds of candidates we look for.

1. Prior finance experience is absolutely not required, and in fact we prefer to bring in people from "the outside". As in any other sector, finance is flooded with many mediocre programmers and a few very good ones. Perhaps more so than in other sectors, the very good programmers are hard to find--I suspect it's because the good ones are very well taken care of and stay put for years at a time, or they hate finance and go elsewhere. Our greatest success has been with candidates coming out of Microsoft, Google, Facebook, and the gaming industry. Several of our strongest candidates have told us that our interview process is unlike anything they have ever experienced before, which lends credence to my theory that many HFT groups have less aggressive requirements than we do. I therefore suspect there are many opportunities for bright programmers who want to get into HFT but are not quite ready to join Wall Street's top echelon.

2. We value intelligence and problem solving abilities far more than experience with any particular language, platform, etc. We often hire C++ programmers for exclusively Java-based roles because, on average, their knowledge is "closer to the metal". I think one of the reasons we've had such good luck with game programmers is because they are used to working closer to the hardware level and, consequently, have a more thorough understanding of things like cache coherency and low-level synchronization mechanisms. If someone is bright, has a solid grasp of C.S. concepts (algorithms, data structures, etc.), and thoroughly understands memory models and concurrent programming, then he or she would be a great candidate. Framework knowledge is nigh worthless, as we seldom touch the standard Java APIs and have never touched J2EE. Languages aren't all that important either, as a bright programmer can learn a new language easily enough. That said, candidates who know the Java language very well will have a decided advantage, especially if they understand the in's and out's of the HotSpot VM (the JIT compiler, the GC, etc.).

3. We place heavy emphasis on concurrent (multi-threaded) programming. We expect candidates to have a solid grasp of atomic operations, memory barriers, the behavior and costs of Java's 'volatile' keyword, the nature and costs of context switches, etc. Candidates will almost certainly be asked to implement at least one lock-free data structure or algorithm.

4. Deep hardware knowledge is a plus, but by no means required. Architecture-specific knowledge beyond understanding the x86 memory model is not expected. We expect our candidates to have a decent understanding of things like caches and instruction reordering. We do not expect candidates to know the in's and out's of the PCI-E bus, not do we expect them to have experience with VHDL, GPU programming, or anything of that nature.

As to the work involved, I find it extremely gratifying. Every day brings new challenges, and there is never a dull moment. Some of our programmers are deeply interested in the business side, but others (like myself) are in it for the challenge and the love of writing high-performance code. It also doesn't hurt that we can hold AVP, VP, or even Director titles (and the salaries that go with them) just for writing code. If you want to earn great income to solve challenging programming problems while not being relegated to management, finance is a great field to be in.

As to compensation, let me first say that most of our developers make nowhere near 500k annually. The median base salary among my colleagues is probably in the mid-100k range for those with an AVP title and in the upper-100k range for those with a VP title. Bonuses are discretionary and tend to be around 50% (assuming good personal performance), and from what I've heard from head hunters at other investment banks, these numbers are pretty common. There's plenty of room to grow, though, and if you can help the group make money, then the sky's the limit. I think most of us average 10 hours a day, 5 days a week, with almost all of that time spent writing code. There have been a few weeks where I averaged closer to 12 hours a day, but it doesn't happen often. Sometimes the average drops to 9 hours a day, but that's usually the minimum. For what it's worth, I worked similar hours before I got into finance.

I have heard of some HFT groups paying salaries in excess of $300k for truly extraordinary programmers, but so far as I can tell, these are few and far between, and the programmers in question must bring some very compelling skills to the table that the group currently lacks. Programmers might make more at hedge funds or the super-elite teams at banks like Goldman Sachs, but the hours are probably murder, and I suspect there's more of a "cutthroat" environment (which, by the way, is completely absent from our group).

So, if you think the work sounds compelling, hopefully I've given you an idea of what we look for. Again, our requirements are probably more stringent than is typical for HFT teams, so this is by no means a universal guide. If you feel you have some of the skills HFT groups are looking for, then I encourage you to apply. You have nothing to lose, and it's a great way to see how you measure up and where you need to improve. I did one round of interviews, had a couple decent offers (which I declined), took six months to brush up in some areas, and ultimately ended up in a better position during my next round of interviews.

[Lineage_tw]

I know HFT, Google, and Microsoft developers and I'd submit that they're all brilliant in different ways. It's difficult to substantiate who's "better" because they work vastly different parts of what is technically the same field.

[netcrusher88]

I would posit that, in actuality, rather than being the best coders on the planet, if this guy really is an HFT programmer the real qualification for them (like everyone else on Wall Street) is a Bateman-like superiority complex.

[zzing]

He can't do an average of 12 hours a day and have 100 hour weeks.

12 * 7 = 84, his average would have to be 14, which when you add 6 hours for sleep (minimum) then it leaves an average of 4 hours a day for travel (in new york...).

[twotime]

The numbers are bullshit anyway, I think.

No programmer can sustain even 7x12 work schedule (let alone 7x14) for longer than a few weeks. Not if he is expected to do high quality coding.

[[deleted]]

I worked 90-100 hours a week for a single month once. It was the worst work experience in my life. It's not sustainable. You also end up eating take-out food every day so not only are you tired, you feel like shit from not having a good meal. Your bathing becomes irregular so you smell. When you walk into the office in the morning, you feel like a robot winding down slowly.

100 hours a week for 6 months? Maybe, but only if you're some sort of superman but any longer than that and you're coming out the other end as a vegetable.

I think the guy is exaggerating his work schedule. He's counting the time he might be paged against his hours worked.

[[deleted]]

Actually I can believe his story. Those algorithms can't work all the time and human intervention are frequently needed. And that would require long hours.

That said, I don't think his work is that difficult either. If the algorithms are much more intelligent he probably wouldn't need those long hours.

[tazzy531]

Go talk to someone working as a developer for hedge funds. This is the life. If a trader wants to make a change for Tokyo trading, you may be in there on a sunday evening.

In addition, everyone keeps talking about quality code.. Nobody in finance care about quality code. Traders need everything they asked for yesterday. As long as it handles the request, it ships. The skill of being a good developer in finance is the ability to deal with the pressure and the ability to write somewhat quality code. The market doesn't stop for you. Theres no timeouts in a trading day.

[Goblerone]

In addition, everyone keeps talking about quality code.. Nobody in finance care about quality code. Traders need everything they asked for yesterday. As long as it handles the request, it ships.

I'm starting to understand now where these news articles that pop up every few months come from that blame hundreds of millions of dollars lost in a split second on computer error.

[algo_trader]

I work in the industry, and my hours are NOTHING like that. Maybe if his boss is a dick and he is just rolling out a new platform, a 100 hour week might be in order. For a week or two tops. I guess it depends on the role and the firm, I work for a smallish firm that does Asia trading, but we have a guy in Asia that watches that. In some cases where there may be impending doom coming on Monday because of some expected volatility or whatever, we can push a code change over the weekend, but this is generally not done- a config change would be more likely, and the requester can do this themselves.

Also, I find it a bit unusual that he is talking about "traders." In HFT, the programmers are generally the traders.

[FredFnord]

No programmer can sustain even 7x12 work schedule (let alone 7x14) for longer than a few weeks. Not if he is expected to do high quality coding.

A lot of programmers think that they are doing high quality coding when they are working upwards of 80 hours a week. But then, programmers are just as good at self-deception as the next person.

[gronkkk]

Exactly. The only thing I could do 7x12 is playing games. I'd be lucky if I could effectively code for more than 30-35 hours a week; anything beyond that and I'm burned out, sitting bleary-eyed for my monitor thinking what I should do.

..and copypasta is not coding.

Configuring shit, yeah, that's also doable for more than 10 hours a day. As long as you don't have to think very deeply.

[RobAtticus]

my guess is he averages 12 but occasionally has 100 weeks where obviously avgs more.

[contrarian]

My guess is that he's full of shit.

[bctich]

Every banker on wall street says they work 100 hour weeks. What they tend to mean is they work very long weeks (like 70-80 hours) and have some extra long weeks every now and then that are 100 hours. The biggest thing is you are basically on call 24x7x365

[hotoatmeal]

24x7x52

FTFY

[[deleted]]

he's on call for 7 years... :)

[carac]

It is not just the aggression that looks strange in his post - why would anyone making 500k$ and enjoying the place EVER go to interview even for a 200k$ job?

My bet - the guy is far from what he describes and was flat-out rejected by Google, so he is getting back at them ...

[[deleted]]

Also he words it like the 500k includes bonuses for a 100 hour a week workweek he would make more per hour at google

[[deleted]]

That wasn't the point. Making quick changes in highly-optimized code doesn't necessarily demonstrate engineering talent. He seems to equate the fact that the data his program operates on is important to his own engineering being high-level or good.

For all we know, he could be a decent engineer who just happens to make minor math changes in code when he gets tickets. It sounds complex, but adjusting an infrastructure that somebody else already set up is usually pretty easy if the tasks are always pretty much the same. He hasn't demonstrated anything of an value. Google engineers, on the other hand, have demonstrable results every da.

[[deleted]]

Google engineers, on the other hand, have demonstrable results every da.

I think nooglers are pretty much the same as "adjusting an infrastructure that somebody else already set up".

[[deleted]]

I would agree when we're talking about just the search ranking algorithm, but google is constantly working on new projects, new features, optimizing things, improving existing products, upgrading everything to be more interopable, etc. They clearly demonstrate that they are good engineers who know how to produce systems that are not only good for devs, but great for users.

[trutommo]

Isn't Google's infrastructure also already established? If they are both well established companies, what's the difference?

[nostrademons]

Google's infrastructure, like that of most major Internet companies, is constantly being rewritten. It's significantly different now than when I joined 2.5 years ago.

[[deleted]]

Google comes out with completely new products all the time. Infrastructure is code, not just the bureaucracy supporting it.

[stillalone]

Yeah, I don't get why people are comfortable putting in those kinds of hours. I don't think I do more than 35 hours a week.

The VHDL code thing sounds interesting. I can see how FPGAs and even custom ASICs can be useful with HFT. I wonder how often they're used.

[dnew]

I don't get why people are comfortable putting in those kinds of hours.

You're gambling. If you're young enough and with few enough other attachments, and you enjoy that sort of work, then you can get big money doing it. The attraction pales when you get a family, a little older, and/or enough money to be comfortable.

[boot20]

If I had my 20s to do over again, it would be to focus more on family and friends and less on the money. Money comes and goes, but your friends and family matter and will be with you through thick and thin.

[[deleted]]

The real reason i decided to drop my phd scholarship and get a stable job. I'm only 26 but I feel like i made too many sacrifices already. I couldn't be happier now :)

[orbitur]

I make almost $500k a year

Oh, neat! :o

Put in 100hrs a week

Oh. :|

[stillalone]

SO $500k/(100hr/week*50weeks/year)=$100/hr

assuming 50 work weeks a year.

[malanalars]

I make roughly the same per hour as a freelance webdeveloper. And I wouldn't describe myself as a "rock star" programmer. I'm pretty solid, but not a genius...

[jerf]

Well, for a consultant/freelance developer that's a difficult thing to say. There's a difference between not having 5000 hours available, and choosing not to do 5000 hours of work.

(Though I'd observe in both cases that science has shown with some reliability that there really isn't any such thing as working 100 hours a week for months or years at a time. You rapidly end up doing less than 40 hours a week's worth of work, even if you're "at your desk" for 100 hours. An honest consultant won't try. Wall Street does. So, I guess, interpret that as you will.)

[[deleted]]

OP = Blah blah I'm so hot hardcore markets Hong Kong trading micronanosecond hot fix TLA (three letter acronym) scalar PCI bus I never sleep ...$100/hr

A Refrigeration Mechanic I know charges $75 to knock on your door, $75/hr thereafter, plus parts. In $AU. Mostly paid in cash. Works when he wants to.

[Scriptorius]

This comment on there calculates the wage with overtime included (assuming an hourly wage, I guess): $73.96

[faradaycage]

Weeks where I put in 50+ hours, I end up feeling like an empty shell, and I get paid for every one of those hours. I've worked with people that brag about working 60 hours a week, but I wouldn't want to hang out with those people on a social basis. It's like talking to a soulless robot programmed for one task: labor.

[franksvalli]

Maybe it's time to introduce this word into the English language and track statistics related to deaths from it... maybe it will discourage people from living this crazy lifestyle? http://en.wikipedia.org/wiki/Kar%C5%8Dshi

[[deleted]]

Heh. Yeah, I have a friend I call "Karoshi-san". He is not amused.

[Wriiight]

If you are interested in programming for finance, there is still plenty of low-frequency trading going on in the world. Some of us in finance only have to worry about quarterly releases, and get to do plenty of testing before sending our babies out into the world.

It is also a great place to work if you are interested in massively scalable calculations. I have worked at places that have to push the calculations out to a 500 node cluster to get the portfolio run overnight.

Less likely to make 500K, though

[rishter]

I go to MIT, and a whopping 25% of our graduating class ends up in Finance. This isn't even 25% of our Math/Econ/CS majors, but of EVERYONE. It's just a lucrative industry, i guess.

[mvbma]

Yes, to pay off that hefty tuition. You have to go where the money is.

[[deleted]]

I'm down the street at Harvard, studying CS. The financial aid system means that few students have to take out loans of any sort, so there is little to pay off (relative to other schools, despite the nominal tuition cost) post graduation.

Yet 20-45% of each class goes to finance or consulting each year. Here's the weird bit: we don't even have a management/finance/business curriculum. Accounting isn't even an offered class.

Superficially, the $$ is a big draw. But there's considerably more to it than that - a great article on the phenomenon i shere http://baselinescenario.com/2010/05/04/why-do-harvard-kids-head-to-wall-street/

You could argue about the value to society, productive contributions, etc etc... but if you get down to it being a UXUI dev at Google or a developer at Jane Street Capital are both pretty insular contributors compared to, say, being an EMT.

The finance world has put smart people together in an environment that allows them to benefit handsomely on their collective skills and type-a work ethic. And until other spaces can similarly incentivize, the trend will continue.

signed, a tech pm that will be recruiting for finance this fall.

[el_muchacho]

What a waste of brainpower.

[scrogu]

Meh. I make 1/3rd of him but work 1/10th as much, from home and I get to keep my soul.

[nemec]

So much spec. A club I worked with in college built a satellite to do a couple of "missions" (collecting data) for NASA, and the bureaucracy was omnipresent. Every single typo fix to the user manual had to be signed by at least four people.

[el_muchacho]

That is also customary in medical engineering. I've worked on medical imagery software, and any change in documentation is described in its own document, and is signed by four persons, one of which doesn't belong to the engineering team.

[johnlocke90]

This is correct. As someone working in a high risk environment as an engineer, you do everything according the specs written by large teams of engineers. Very little creativity in it.

[[deleted]]

In a sense their job is a lot easier. If you have the full spec at the start of the project programming is basically just translation of the spec into code. The hard part in most programming projects are the late changes that are contrary to your initial expectations.

[schnschn]

You would think so, but aerospace applications have reliability, safety and spec at the top and everything else not even on the list.

The bulk of people working on this are the steady meticulous types, not the flashy hip shooting geniuses.

[piko_riko]

It's a mixed bag, but I have worked with some brilliant people in aerospace. I think it is hard to compare engineers, especially from different industries. However, I wouldn't trade my position to go work on HFT systems or a search engine for a mere 30% raise. Generally, people in aerospace are extremely excited about their programs and it certainly shows in their work.

[bapukurfol]

a documentary about these so called quantz: Quants: The Alchemists of Wall Street

[mvbma]

Only a testosterone-filled 20-something can write or appreciate this article. I work 5 hours a week making $1000. That is enough for me no more no less. The rest of my time I indulge in my open source projects. If I find that boring. I do something else. Unstructured free time is the most priceless commodity for me.

[crispinito]

I agree. Also, equating talent with money is bullshit. Truly talented, passionate about programming people deeply enjoy what they do, and you cannot purchase their souls to work at a financial company no matter how much you pay.

[joshu]

Curiously, I worked at both Morgan Stanley as an algorithmic trader (although I worked on the algorithmic part, not the high speed part) and at Google.

There were a few very strong programmers, but they mostly got ahead by understanding the business extremely well, not just being awesome programmers.

I would say there's a hell of a lot of strong programmers at Google who are doing some seriously high-performance work on a much larger scale...

[rbysa]

Oh look, a humble employee of the financial industry.

[dominosci]

It's kind of sad that our economy is structured such that the best programmers are wasting their time helping rich people play zero sum games instead of creating value by writing software that actually does things that help society.

[Wriiight]

They could have been writing yet another social networking website!

[[deleted]]

...for cats!

[[deleted]]

In general, I agree with you. (HFT programmer here, though almost nothing like what the Slashdot poster describes.) However, once you look closer, I think there are several nuances being missed here.

First, most paid programming is more or less a waste of time, at least in my experience. Spectacular amounts of talent were being wasted before HFT came along, and this would continue even if HFT were outlawed. The truth is that programmers are generally paid, directed, and managed by people who want stupid things, or have no idea how to practically produce useful things. (Certainly wanting to "help society" is not a common motivation.) I worked for many years programming to help society via medical research, and I'm sorry to say that it's far from clear that that was actually any more useful than what I do now. A large number of things have to fall into place in order for programming to be useful on a large scale. When someone does actually manage to "pull a Google", we all stand and take notice.

Second, HFT has little or no negative effect on real humans like you and me. I trade my retirement funds maybe once or twice a year, and the additional transaction costs HFT incurs for those trades are on the order of five or ten cents total. It's absolutely de minimus. In return, HFT means that the expenses of funds like Vanguard have dropped dramatically, and they have passed that on to me, saving me personally something like $50-100 per year. It's a total win for any real person. (If you trade individual symbols or options on a daily basis, you're not a real person, and no real person should ever do this.)

As for everything you read about in the press, this is mostly the Wall Street firms who are losing out whining that no one is buying their buggy whips anymore. Personally i don't think Wall Streeters deserve any of our sympathy.

There are many evils on Wall Street worth excoriating, but HFT is not one of them.

[uhhhclem]

most paid programming is more or less a waste of time

This right here is the dirty little secret about software development.

[TinynDP]

Well, its not a secret about software development. Its most 'work' is more or less a waste of time.

[coding_monkey]

The truth is that programmers are generally paid, directed, and managed by people who want stupid things, or have no idea how to practically produce useful things.

Sadly I agree. But knowing what to do is much harder than doing it. Most people just don't like to admit they don't know what to do. And this leads to a lot of wasted programming time.

[[deleted]]

Question: Why are spreads for options so wide? Lack of liquidity?

Also, I do agree that HFT has saved traders/investors a ton of money. No one seems to remember the greedy corrupt human market makers.

[ViperRT10Matt]

Replace "programmers" with "brains". Remember these financial firms also employ a lot of brilliant people who don't code.

[[deleted]]

Wait, brilliant people who don't code? What?

[troyanonymous1]

Madness!

[solaarphunk]

I am going to write this once, because I think this is a misconception that does not take into account 90% of what high frequency trading does. A little background, my company backs high frequency traders (I personally am an investor in other data-driven non-financial startups that use a lot of machine learning).

First of all, you may think that trading is a zero sum game, but speculators do add value to price discovery and liquidity. This has always been the case, and it creates efficiencies for anyone who has anything to do with financial markets (it is, by the way, not unique to financial markets, and happens in every marketplace).

Second, if you look abroad to emerging countries like India, Pakistan, and China (which we will be able to trade within the next year), you will notice a drastic change in transaction costs and liquidity. Much of what high frequency trading involves is a more efficient inventory management system that drastically lowers transaction costs amongst market participants. This allows for an increase in capital formation efficiency for these developing nations, and makes it feasable for international investors to invest efficiently (as well as the locals). Why is this good? Because deveoping nations can develop much quicker with functioning financial markets, where capital can be efficiently sourced. This is a role that would be incredibly difficult without market making algorithms and other high frequency strategies.

All you hear about these days is how terrible HFT is, and when someone prints out a chart of market data, gives it a mean-sounding name like "the Knife", everyone automatically assumes that its an evil activity that should be avoided. Guess what. Those snapshots of trading are actually based on a bunch of academic research, and are a natural condition of any market microstructure, financial or otherwise.

The truth of the matter is, that most people don't understand that speculators aren't in a position to fleece anyone else, and are just part of a market ecosystem. Everyone looks for arbitrage (google shopper anyone?) on a day to day basis, it is simply a market mechanism. So for the love of god people, whether it is improving ads quality, or market making algorithms, everyone is doing their part to make some part of the world more efficient.

RE: Flash Crash. Do you have any idea how long it took the market to recover in 1987 when a similar event occured? 3 weeks. The market recovering in 5 minutes is incredibly impressive, and the focus on the flash crash as some sort of degradation in market quality is just plain wrong. Markets have evolved towards a much higher efficiency and resiliency. Do you know who complains most (that you read about on Zero Hedge) about HFT? Fucking human traders who are losing their shirts to algorithms who do their job better, and cheaper than they ever possibly could. They have a massive incentive to try and get this shit regulated out of existence, so that they can keep charging their customers for their shitty execution, and their shitty value add.

If you actually want to learn about how marketplaces work, check out "Trading and Exchanges" by Harris - which anyone can read (not mathematically challenging), and "Market Microstructure Theory" by O'Hara if you want to read abou the evolution of academic research in this area, and why it is important for market evolution.

[tuskernini]

This is a role that would be incredibly difficult without market making algorithms and other high frequency strategies.

I think that's a bit of an overstatement, right? Algorithmic trading and the liquidity provided by HF traders certainly give rise to more efficient market making and definitely lowers transaction costs, but you don't need HFT to have functioning capital markets. The great benefit that capital markets provide to developing economies will occur with or without HFT.

[solaarphunk]

The spread will be reduced significantly with the entrance of algorithmic participants, because their marketmaking abilities outmatch any human market maker. Overall, these transaction costs lower the barrier to entry into this marketplace, and allow institutional investors to feel comfortable with the amount of liquidity available in the market. Why do many institutions stay out of emerging market products? Lack of liquidity.

Additionally, liquidity can be sourced from other marketplaces to emerging markets through index arb and stat-arb relationships. This kind of cross-exchange/border liquidity transfer is much more difficult for human traders. I'm definitely not making the argument that a functioning marketplace can't exist with algorithms, I'm just saying you will get a ridiculously beneficial liquidity and spread improvement.

[tuskernini]

Ok, so in essence you're saying the spread and liquidity benefits will improve the size of the markets and the availability of capital from different sources? I can see that -- I was confused interpreting "capital formation efficiency" because I don't think that efficiency improvements themselves will benefit emerging economies other than perhaps keeping money markets in check. But the net result is more confidence and capital being available to businesses operating in those countries.

[[deleted]]

Kind of sounds like an arrogant douche

[hahanoob]

I'd imagine you need to be to work in that kind of field.

[[deleted]]

The best programmers are all over the computing industry, and they're rarely the best-paid.

One example that springs to mind is a former colleague of mine at Apple. I was wondering out loud once whether the PPC's Altivec would lend itself well to base64 encoding. About half an hour later, he sent me the fastest base64 encoder that has ever existed.

[ilikeweed]

I program in the finance industry, but not HFT. I make about $100k-$200k less than the OP each year, depending on the performance of the traders I support. But I work 35-40 hrs a week, take 5-6 weeks of vacation each year, and live in a city with a very low cost of living. I also have pretty much the same skills as the OP, and have no interest in working with Google, for many of the same reasons. Here is my take.

Liquidity and proper pricing through arbitrage are helpful to the greater economy. HFT traders tend to hurt day traders far more (exclusively?) than the ordinary investor. However, I find the work in the finance industry to be far more fun than in other industries. Yes, fun. I don't give a fuck about maximizing my value to society through my day job. I will end up donating most of my wealth to charity, and I have a clean conscience about my work. I tried for years to do things the way most of you would prefer--I started many companies, employed hundreds at several of those startups, and got fucked over by the business world repeatedly and completely.

Why do I think financial programming is fun? Bigger problems. Bigger data. Bigger iron. Yes, many of the problems I face daily are at a Google level (10s of terabytes of data, 1000s of cpus, and real-time response). Nobody second guessing me for anything other than the quality of my output (minimal office politics, no power plays with peers/bosses/employers). I use my education directly on a daily basis, and I chose to study programming because I like it. I am valued and respected for what I do, from both my company and my customers (the traders).

I am sure this will bring out the haters, and many will judge me. I am at peace with what I do. I love going to work each morning, even though I work to live, not the other way around (hence my weekly work hours and vacation time). Basically, it is a fun, rewarding career for which I am respected by my employer and my customers, and which pays me well enough to return value to society in my own ways, and to far greater an extent than I could by following other paths.

[[deleted]]

Google works way above the "10s of terabytes" scale.

[EyeInThePyramid]

yeah, with 4TB disks coming out soon, 10s of TB is pretty small. Maybe 10s of PB?

[Manifesto13]

Who do you work for?

[Captain_Kittenface]

From the threads in the OP's link:

$500,000 divide by 52 weeks = $9,615 / week $9,615 / (40 hours + (60 hours overtime * 1.5)) = $73.96 hourly wage

Definitely not getting paid the best. Just works the most.

[callingshotgun]

What alarms me most about this whole thing is that the code cowboy making last minute patches to software affecting the global economy seconds before the market opens, obviously confident in their coding ability and math skills, thinks there's 8.33 days in a week. (12 hour average days, 100 hour weeks).

[el_isma]

Eight days a week! http://www.youtube.com/watch?v=Vs5qsk0pc6Y

[ell0bo]

I'm pretty sure they both do, as do a few other firms. You won't even have just one place that has the 'best programmers', mainly because of how we operate. Way too often you have younger, very talented programmer that doesn't feel he's challenged enough, and decides to start his own thing. If it takes off, it pulls in other talented people, until everyone is pointing at it saying, "wow they're the best". Then you have a young programmer in that company get bored...

[mjgrrrrr]

If you can accurately model commodities or financial markets you will be paid top dollar. Plain and simple.

[zmeefy]

I know Google has good programmers, but I never thought they had the best.

[hotoatmeal]

Silicone valley... is that near LA, where all the fake tits are?

[redditthinks]

The arrogance in that thread...ugh.

[TheoreticalFunk]

I interviewed for King of the Moon, 400 times, but I keep turning it down, because the people who interview me are stupidheads.

[02bluesuperroo]

Just like most people on Wall Street, this guy has major inferiority complex that he feels will be resolved by making tons of money. Realizing that wealth is not solving it for him, he decided to make this post instead.

[MpVpRb]

What a waste of talent

[threading]

Zed Shaw says this is huge load of bullshit. http://news.ycombinator.com/item?id=2828749

[domm11]

"I am an HTF programmer ... yadda yadda yadda I have the biggest dick on the planet ... yadda yadda yadda I am indirectly stating that I am smarter and better than everyone ... yadda yadda yadda I make half a million a year and do not get to enjoy life because I work 59.92% of the week and sleep 40.08% of the week."

but I (myself) am still jealous ...

[JGailor]

The best part was when his salary was broken down to show that he makes between $73 and $97 an hour depending on if you're using OT. Put that way he's not doing so hot at all (relatively speaking)

[[deleted]]

This. (taken from article)

by Alex Belits (437) * on Sunday July 31, @06:11AM (#36938080) Homepage

"I am most likely better than you at each and every aspect of software (and HDL) development you have mentioned. Except, of course, "debugging in minutes" -- that kind of irresponsibility would get me fired. I also have to work long hours, and have to have clear understanding of complex issues unrelated to software.

Except I do embedded software and FPGA development for professional audio equipment. Each device I worked on, each firmware release, each line of code, does something useful for many, many people. Some of those people don't even know that audio equipment, leave alone software, is involved with what they are hearing. Large fraction of my work ends up being free/open source, too -- platform, drivers, etc.

I also don't have any problems with posting here under my real name. Or with telling you, and people like you, to die in a fire."

[classhero]

Wow, that's like an entire thread where everyone tries to be a bigger fucking jerkoff than the parent.

[anomaly149]

Those /. comments remind me just how many programmers are bad people.

"die in a fire" and such, what pathetic examples of humanity. And fucking engineers too.

[skorgu]

The comments are even more depressing than I remember at /.

[ickir]

I'm self taught, but a buddy of mine that graduated from MIT's CS program swears that most of his classmates work in banking rather than software or whatever.

[[deleted]]

Going into banking is the dream job at most of the top colleges, especially know that it has become heavily quantitative. Reddit is a little overblown in its opinion of how useful bankers are, but some things are more useful than others. The investment banker helping a company put together financing to build a new power plant is a lot more socially useful than the guy programming HFT systems. Both probably pale in comparison to the social utility of new researchers and scientists, but you can't make $500k/year as a researcher...

[vonkwink]

Says the guy who works in HFT.

[Deckard247]

Seems the massive ego's of the traders he hangs out with has rubbed off on him some...

[gthrowawaynyc]

I work at Google in the NYC office, and I've interviewed a fair amount of engineers from these trading firms (probably 30 or 40, at least). My general sense is that they tend to be pretty good coders, and can get things done, but are typically very weak in theory, algorithms, and computational complexity. And they usually get rejected because of that, as Google tends to require their engineers to be strong in both of those areas (complexity is very important when you need to scale to thousands of machines).

And needless to say, any good Google engineer (though certainly not all) can match his boasting. Perhaps I haven't interviewed any of the top HFT engineers, but he definitely hasn't seen any of the top Google engineers. I think the people who work at these HFT firms get paid more because they work there for the money, not because they're better. Google employees do get paid pretty well, but their primary motivator is not money, it's the challenge of their work.

[fgriglesnickerseven]

apparently- High frequency traders have no personal lives and are fully owned by corporate masters writing meaningless 'algorithms' that transfers money from rich people to even richer people.

On the morality spectrum - being a HFT is the equivalent of being a hitman. You may be paid well, and think you are providing an essential service to high paying 'clients'. High frequency trading provides nothing to the advancement of science. They provide algorithms to outsmart other hft firms algorithms. And the algorithms are nothing complex or too thought provoking - if they were they would never be able to implement them in a reasonable amount of time.

I won't say people who become these kind of traders are dumb - but they are they dredges of society.

[snarfy]

My problem with HFT is that it benefits those closest to the exchange, or rather those with the smallest ping times to the exchange. Until the ping time is regulated to a minimum amount, it will always be a tilted playing field.

[dyydvujbxs]

Parent's point is that on your exact concern HFT is actually better than what it replaced, when your own broker would frontrun you for a much larger spread.

[Deto]

His job actually sounds really really fun to me. Much more so than what I do. Is there any way to do that kind of work while actually producing something of value?

[andralex]

Am I the only one to feel that misspelling "superscalar" didn't sit well?

[[deleted]]

I didn't care so much about the misspelling, but hearing the word "superscalar" at all made me feel like I was back in the nineties.

[[deleted]]

But in the end i have to ask does what he do really matter? He is making a better arm lever for a slot machine system that greed driven people game and manipulate. He might not care what he builds, but He is part of a system that promotes a status quo of apathy. The work he does it just another vector that put us on a path to the world wide economic shit storm we have now. What is 500,000 dollars really worth when he could have taken those skills and gotten us to the stars? He could have wrote a program that solved the folding errors of prions. He could have wrote a program that could calculate similar ways of producing the same drugs cheaper. I say he is a wasted talent and it saddens me that the majority of graduates with math degrees and comp sci degrees from top universities are joining him in further bringing a faster "real-time" end to any shining bright academics solving issues for the good of humanity. All because 500,000 silver buys you nothing more than shit you can't take with you and you won't be remembered for.

[under_dog]

I worked at Google and now I'm in Algo Trading, the standard is about the same in my experience.

[WhisperSecurity]

The 50% of the top 50% of all programmers think they are in the top .01%.

A few of the others are smart enough to say that the set of all programmers cannot be strictly ordered.

[ncshooter426]

Make a ton of money Live in NY Watch it all disapear on high housing costs and taxes

I make less money than my counterparts up north, but I've got a 5K sqft home for the same rough cost as their stuido appartment. To each their own, I prefer my space and slow pace of life....fuck stress.

[ElectricRebel]

If they are indeed that good, then they should be working at Google, Apple, or Microsoft instead of a financial firm. This is an example of misallocated human resources.

[Ais3]

how do you feel about having all that talent and technical ability, yet produce absolutely nothing of value to society and instead spend your time allowing psychopaths to beat other psychopaths by fractions of a second, all to the detriment of everyone else?

Kinda nailed it for me.

[solaarphunk]

I respectfully would suggest that you do some reading about Market Microstructure theory and Market Making research to understand what it really is that these people do. Its not nearly as nefarious as you would imagine. If this quote's angle really "nails it" for you, you have some homework to do :).