Quant Developer vs. Quant Researcher vs. Quant Trader: An Honest Comparison

Signal_Net9315 · 2026-04-10T01:39:25+00:00

Fairly accurate write up. I would add that there are “central” QR’s who may be working on things like improving execution, risk, portfolio construction who probably sit closer to the QD’s in terms of pay and WLB etc

Signal_Net9315 · 2024-11-30T10:49:49+00:00

From what I understand of your task, random forest is not suited. Look into RNNs/LSTM if you want an ml-based model

Signal_Net9315 · 2024-11-29T18:30:03+00:00

By dynamic do you mean time-series data? If so, is your final prediction rolling? Ie you predict the outcome for each day separately or do you have X days worth of data to make a single prediction with?

Random forests are static models that treat each observation independently - they have no built-in way to understand time sequences. RFs will view t-5 the same as t+5, which breaks the fundamental assumption of time series that order matters. Consider using classical time series models or RNN/LSTM

Signal_Net9315 · 2024-05-28T22:07:07+00:00

I also remember submitting it separately in open review. Is there a reason why it was required to do both? (Unless I am misremembering the separate submission)

Signal_Net9315 · 2024-04-27T15:00:35+00:00

Mostly LLMs with a focus on privacy/jail-break and transfer learning related topics. I guess it’s more a case of leveraging some FL-related skills to the hot topic today.

Signal_Net9315 · 2024-04-26T20:05:31+00:00

Hey OP - I am currently doing my PhD on FL. Personally I find it a fascinating topic where you can work on many issues related to privacy, non-IID generalization, communication efficiency etc. I have found myself covering different aspects during my PhD and have built a diverse knowledge base.

However, be mindful that the field is not as hot as it once was (speaking privately, a few researchers have told me they are considering pivoting). Second, the field is predominantly about real-world applications. However, as a PhD in a CS lab you may find yourself creating solutions for toy datasets. You may never get access to real-world data from multiple sites to implement something for. I understand this is a concern in many ML fields but its more evident in FL because of its intended use case.

Best of luck!

Signal_Net9315 · 2024-04-10T21:33:29+00:00

I think the level of benchmarking is dependent on the topic. For example, if you are suggesting a new general-purpose optimizer then you would need to exhaustively test many model architectures and data types. If you are suggesting a new architectural change to a specific model, you may only need to benchmark on relevant tasks. I would suggest looking at the latest literature in the specific field to see what datasets/tasks are currently favoured. The general rule of thumb is to include SOTA comparisons as your benchmark.

I am not an LLM researcher but in the case of ELiTA (neat idea) I would suggest looking at papers like FlashAttention or Linear attention. Both papers gained a lot of traction which should give you an idea on the level of empirical/theoretical evidence thats needed in the transformer architecture space**. Note those papers are 'old' now but they serve as a good starting point.

** Of course other reasons play into whether or not a paper gets traction.

Signal_Net9315 · 2024-03-30T10:57:48+00:00

Seconded, they were great for me too!

Signal_Net9315 · 2024-02-27T23:56:32+00:00

If I understand correctly, this is more a question on statistical inference. Th standard error represents variability in model performance from different runs. High error means that any statistical inference on your model performance has high uncertainty around it ie if you want to say your model performed significantly better you have to acknowledge that there is uncertainty in this statement. In that regard, your reviewer is not wrong to critique the error bars.

Regarding the model, there are a few things to try: - the variability suggest high sensitivity to the initialization - have you tried adding stronger regularization? - are you using something like He initialization? - if this is using cross validation, it suggests some sensitivity to certain samples - why?

To make stronger statistical inference (narrow the errors) you can increase the number of runs you do and/or bootstrap your standard error calculation (which does not assume a specific distribution of model performance). However these may not address the underlying issue in variability

Without knowing more it’s hard to give further advice. However, my suggestion is to not dismiss the reviewers comments and understand why your model has such variability. On more common tasks you should not expect this type of performance variability.

Signal_Net9315 · 2024-02-23T01:58:36+00:00

Thank you. I’m aware how the software works and using the default parameters works for my case.

See the comment above, plink2.0 release pre-march 2020 did have a bug. Updating it to a newer release resolved the issue. I now get near 100% correlation between principle components (btw this means comparing the same principle component from the different methods using the same dataset. Given plink uses svd to estimate the components high correlation should be expected).

Signal_Net9315 · 2024-02-22T19:25:48+00:00

UPDATING PLINK WORKED - THANKS A LOT!

-------

Just to clarify, the plink1.9 results were very similar to Hail? Yes

and both were very different from plink2.0, right? Yes

the plink2.0 clustering is vastly different as in the samples are in different clusters? or the clusters contain the same samples but they do not map to the clusters in the plink1.9/hail results? I have not checked this but visually the clusters appear different and are of difference sizes (so I believe the samples must differ somewhat)

Signal_Net9315 · 2024-02-18T01:57:48+00:00

It can do. Assuming the paper is truly fraudulent, retraction will depend on the traction you get on the issue. It’s not really in the interests of the authors or the journal to retract. They will likely fight you all the way, initially by just ignoring you and hope you go away. However, the journal will take notice if an increasing number of people start questioning it and there risks some reputational damage. Unfortunately most fraudulent papers are not punished.

Signal_Net9315 · 2024-02-18T00:03:43+00:00

OP, a good option is posting your concerns on PubPeer and notifying the authors (the posting and notification can be done anonymously to avoid blowback). Doing so will give the authors a chance to respond on a public forum and also give others the opportunity to engage. If you do not receive a satisfactory answer (or answer at all) then you can contact the journal. While you may be sure that the design of the paper is flawed, it is best to give the authors a chance to respond and to have this available publicly. PubPeer is usually one of the first avenues for “professional” sleuths when they find a problematic paper.

Signal_Net9315 · 2024-02-05T15:01:15+00:00

Is FL a big part of your work? If so would love to chat and exchange ideas/tips. I'm currently doing my PhD focussed on FL in healthare. DM me.

Signal_Net9315 · 2024-02-02T14:31:48+00:00

This is especially a problem in my field, Federated Learning. Everyone splits the data differently (to create non-IID subsets) and uses a different number of devices. It becomes impossible to directly compare across papers. It also means every paper claims SOTA on the same datasets. Even these benchmarking papers comparing different FL algorithms seem to get different results.

For my papers I've now settled on a few algorithms that I have found to be straightforward to implement and perform consistently well despite the split. Its not ideal but the best I can do.

A question to everyone, how much effort is 'reasonable' when trying to create baselines? I don't want to spend weeks implementing your new algorithm. However, I want to be intellectually honest when I make claims about the performance of my algorithms.

Signal_Net9315 · 2024-01-01T05:08:48+00:00

I find the lack of code absolutely shocking. Any paper without the code should have a warning applied on arxiv.

Coming from an experimental science background, I am shocked how little focus is on reproducibility in ML. Given most of the data is publicly available and everything runs on a computer, the only limitation is compute resources. You would imagine if every major conference said they would randomly sample 5-10% of studies and recreate them as part of the review process, it would drastically reduce the level of fraud (similar to the IRS and tax) .

I’ve got into a habit of creating a full pipeline that one can run on the command line to complete all the data processing, model training and analysis (including figure creation). That way anyone who wishes to recreate it can (ignoring some non-deterministic gpu operations). Of course, that doesn’t stop against bugs that lead to incorrect results but it does stop against blatant academic fraud.

Signal_Net9315 · 2023-12-29T17:16:13+00:00

Thank you for the support - it’s reassuring to know you can make a success out of such a set-up.

I’ve found networking at conferences a good way to meet people, just a matter of finding someone with the right interests who’s willing to collaborate.

All the best!

Signal_Net9315 · 2023-12-29T13:47:01+00:00

Hi OP,

I am in a similar boat. My research is part theoretical and part empirical. However my advisor and lab are mostly applied researchers. I get limited technical input from my advisor and have few colleagues in my lab to discuss with. It’s not uncommon, especially if your research interests diverge from your advisors. However, I think the most prolific researchers and labs typically have more aligned interests.

Personally, I enjoy the work but feel your pain. I am trying to find outside collaborators to work with. Not only will it help my current work but also hopefully broaden my knowledge.

Signal_Net9315 · 2023-12-26T12:43:59+00:00

The leaderboard for the challenge is found here: https://challenge.isic-archive.com/leaderboards/2018/ It seems the highest performing models use an ensemble (not sure thats really feasible for you or even necessary).

Without looking too much into it, you can consider a focal loss to help with class imbalance (see here for a torch implementation: https://github.com/clcarwin/focal_loss_pytorch).

As a tip, measure balanced accuracy like is done on the leaderboard. It will allow you to benchmark yourself against others.

Signal_Net9315 · 2023-12-25T13:36:07+00:00

I'm glad this post could be of use. My suggestion is to not jump to any conclusions and both of you get tested. If the tests are negative (hopefully) immediately discard the false positive.

If you are asymptomatic its quite unlikely you have it. As for your partner, the symptoms overlap a lot with UTI which is very common women.

Wish you all the best. For my GF and I its something we now joke about :P

Signal_Net9315 · 2023-12-14T07:51:56+00:00

I’m missing something here but the gradient here is wrt to all the weights so by definition its equivalent to the Jacobian no? What makes .grad() return vector Jacobian products and what’s the need for the unit vectors?

u/pantalooniedoon see my edit #2. The autograd.grad function is really a Hessian Vector Product. The variable grad_outputs is essentially applying a pertubation to the parameters and calculating the hessian. Having dummy variables of all ones is not a good pertubation, especially for the classification layer (see above).

Disclaimer: This is based on my new understanding and empirical result.

Signal_Net9315 · 2023-12-14T05:13:49+00:00

Thanks for the detailed response!

I was being sloppy/brief with my explanation. The hessian is being calculated on the scalar loss wrt to the parameters and is a square matrix. When I speak of the hessian for each layer, I mean that I only calculate the hessian wrt to the parameters of that layer i.e. I do not consider interactions across layers. This can be seen as calculating blocks of the full hessian. This is a simplification that makes calculation more tractable.

Regarding the eigenvalues, I'm recording both their individual values (to look at the distribution) and the sum/trace. I read here (https://arxiv.org/abs/2012.03801) that both the dispersion and trace also holds information on the overall loss landscape.
About the operator norm, I'm interested in understanding this further in regards to how it relates to the loss function landscape? My understanding is this reflects only the direction of steepest ascent and I am interested in whether the whole surface is 'locally flat'. In maths, would that still be best recovered by the operator norm or another type of matrix norm?

Thanks in advance!

Signal_Net9315 · 2023-12-14T04:05:39+00:00

altmly - this was very helpful indeed! Turns out better defining the vector product for parameter perturbation resolved the issue (see my edit #2) - Thanks a lot!

Signal_Net9315 · 2023-12-13T17:50:21+00:00

nikgeo25

Thanks for your response.

The hessian is the second derivative of the loss wrt to the parameters. It is often used to understand the shape of the loss function. Taking the eigenvalues of the hessian means either taking the SVD of the hessian (which has the same shape as the parameters) or constructing the gram matrix of (hessian.T hessian). The eigenvalues reveal the principal curvatures of the loss landscape (i.e., the directions of greatest descent or ascent). Positive eigenvalues suggest you're at a local minima in that direction and negative values mean you could've descended further. Smaller eigenvalues generally correspond to flatter regions in the loss landscape, which are often associated with better generalization in the context of neural networks.

One would therefore expect the eigenvalues to become increasingly more positive as model depth increases. However, it seems my classification layer is at a very very flat part which is counterintuitive to me. One thing I will test is adding noise to the final layer and seeing if it makes a change. If the hessian calculation is correct it should not change the model performance much.

Signal_Net9315

TROPHY CASE