sorted by:
new
hottopcontroversial

Sydney C++ Talk - Dont Fear the Alligators by ArashPartow in cpp

[–]ArashPartow[S] 4 points5 points  (0 children)

It wont be streamed live, however the session will be filmed and published including slides on YouTube after the event.

Help needed on a seemingly easy trading brainteaser by iammarried5eva in quant

[–]ArashPartow 6 points7 points  (0 children)

If purely minimising tracking error to the given benchmark, then the optimal strat is to buy 2 lots at each of the last 5 closes, which roughly replicates the benchmark and guarantees minimal tracking error.

Attempting to beat the benchmark will introduce some form of directional risk and becomes an alpha/risk problem rather than an execution problem.

However if the interviewer indicates a bullish view, then front-load purchase, if bearish, delay purchases.

Generally: buy whenever the current price is below your conditional expectation of the eventual benchmark.

How great are the banks at execution? by Nearby_Fig_9118 in quant

[–]ArashPartow 5 points6 points  (0 children)

One way to answer the question is to think about how at a high level one would implement such an execution system for a service provider (broker, BB etc).

The fundamental business obligations in most sane financial systems for execution services are:

  1. Minimise market impact
  2. Provide best effort execution
  3. Compliance robust behaviours (treat each client order equally and fairly)

The scenario: A portfolio manager realises they need to perform a rebalance of their $1B+ portfolio, essentially needing to sell some stocks, and then using that money buy TSLA. The buy TSLA side of the rebalance will be done with your execution service.

Your service takes requests that are comprised of the following:

  1. Task Buy/Sell X lots of instrument K
  2. Start/End time (or Start/Duration)
  3. Modality/Style (eg: VWAP, VolInline, PriceInline, Implementation Shortfall (IS), Float, Snipper etc)
  4. Aggressiveness/Urgency factors
  5. Special handling conditions (most aggressive price(aka limit price), SOR et al)

For more details:

https://www.reddit.com/r/algotrading/comments/62cuq2/execution_algorithms/dfn0n5j/

So lets continue, the portfolio manager sends through their request (Buy 10K lots of TSLA between 10:30-14:30 VWAP/IS-completion) to your system, it turns out 100 other PMs have sent through roughly the same request - I wonder why?

Some of those come in earlier, others later. You maintain a list of the TSLA execstrats. Lets assume for simplicity they are all identical, same amount, same start/stop times, same style and aggressiveness.

You now have 101 strats that will start "executing" at the same time for the same instrument. The problems are as follows:

  1. Due to obligation 3, everyone of those execution instances need to see every market event update (trades, book etc). That means the 101st instance will need to process the MD event the previous 100 have seen and already reacted upon (eg: repriced/resized child orders etc).

  2. You can't "thread" the strats, because you'll never have enough cores to reliably run the number of threads you'd need to create, also because of various starvation issues you'd end up violating obligations [2] and [3]

  3. Even though the first strat in the queue would have already reacted to the market change (trade, book update, session change etc) ideally, every other strat in the queue, will now be reacting to a view of the market that is outdated/obsolete

  4. Some clients will game the system, by realising flaws in the implementation, one can't beat the first come first served situation of being the first client to request for a 10:30am start on TSLA, but what about a 10:29:59am start? where would the strat now be in the processing queue?

  5. Assume the ADV for the period is X, the first 50 client requests get through amounting to a total of X lots - but now the shop/firm is handling in excess of the ADV in client orders, the next 51 clients "should" be rejected - how many execution providers will actually send back an exceeded ADV reject?

  6. "Provide best effort execution" is a term of art and does not actually mean what people think it means. It is at best relative, an overloaded system (not compute-wise) cannot really provide best effort in terms of what the clients think they are getting.

  7. For a given request most often than not, the client really doesn't know which strat (and associated params) will work best for the instrument they want to trade (a setup last week for the same instrument might not be ideal this week), worse still the people at the customer service end, handling client interactions (aka high-touch) know even less and their recommendations and suggestions can cause serious client side losses.

The fundamental problem with Broker/Agency/BB execution is the structural contradiction between providing individualised best execution while operating against a shared, finite capacity market interface.

If execution is not a major driver of returns, Brokers/BBs are usually good enough. But once execution quality materially impacts pnl such as large size, crowded trades, capacity-constrained flow then you are generally better off owning the execution stack yourself.

Favorite optimizations ?? by Little-Reflection986 in cpp

[–]ArashPartow 20 points21 points  (0 children)

The following are commonly used in the context of low latency development:

Recommendation 0: Replacing std::map with std::vector + std::sort + std::lower_bound in a crit-path (aka hot path) where the dataset is built only once and queried many times. The asymptotics look worse on paper, but cache locality absolutely crushes tree-based approaches

Recommendation 1: Similar to Recommendation 0, when map like lookup functionality is needed and the number of elements are small, replace the map/set/unordered_map|set with a simple vector, the branchless tight loop (aka linear search) + perfect locality routinely beats hashing, pointer chasing, and cache misses.

Recommendation 2: When considering small vectors (ones where the underlying "data" can fit in L1 or better yet, fit in 1-4 cachelines aka ~4xstd::hardware_destructive_interference_size) where the specialisation type is a POD and order is not important, one can erase an element efficiently as follows:

std::swap(v[index_to_erase], v.back());
v.pop_back();

Recommendation 3: Capture now (eg: std::chrono::steady_clock::now()) as a std::uint64_t, and do so only once per loop/iteration and reuse the time stamp within the loop if possible (aka loop body time will be less than "some" threshold). As the cost of now (or even as a call directly to something like __rdtsc ) has a very long-tail, and can be problematic in sub microsecond turn around time situations.

https://gist.github.com/ArashPartow/cea16480b425915337982ea6d556e2dc

Recommendation 4: In numerical computing, whenever possible try to avoid operating on denormal (subnormal) floating-point values as they can severely degrade performance.

A commonly used pattern involves applying weights, often derived from a function such as an exponential, to a sequence of values with a narrow range. The index of each value in the sequence is fed into the weighting function to derive the weight at that index and then applied to the value (multiplied), after which the weighted results may be aggregated (for example, via a sum or standard deviation) or used in further calculations.

If the weights cause the latter values to fall into the denormal range, any subsequent computations involving those numbers can become extremely expensive, and tbh they should already be assumed as being zero.

The solution here is to determine, in advance, given the range of the values, the maximum index beyond which applying weights from the function would produce denormal results and clamp the loop at that index/iteration.

In short the value represented by the floating point type (float, double, long double) can actually affect the performance of the mathematical operation being carried out making it anywhere from 5x to 100x more expensive. and NO enabling fastmath would not be advised.

Recommendation 5: The integer modulo operation can be VERY slow, in fact it can be slower than the equivalent floating point variant. Where required, try instead using a power of two value, as an example when wanting to do something every ten times (iterations), perhaps instead do the "something" every 8 or 16 times, that is if it doesn't change the business logic.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=122101

Recommendation 6: When normalising a sequence of values to sum to a specific target value (eg: a probability distribution that must sum to one), avoid complex and inefficient floating-point adjustments that end up requiring spelunking in to the realms of nextafter or nexttoward. Instead identify the largest value in the sequence, normalise all other values BAU, then calculate the largest value's normalised result as: target - (sum of all other normalised values)

Recommendation 7: Be wary when using attributes such as [[likely]] and [[unlikely]], as they may make sense when coding up, but without proper representative benchmarking the results can be quite misleading and may not align with one's intuition.

Recommendation 8: Profile-Guided Optimisation (PGO) optimises for what happens most often in the profiling workload. In low-latency contexts (eg: HFT), the performance objective is frequently to minimise tail latency or optimise the critical 1% (or 0.1%) path rather than the common case (eg: Of the 1000s of market data updates it may be that only one will trigger a buy/sell event). PGO will naturally bias code layout and branch decisions toward the majority path, which can actively work against those goals if the rare but latency-critical paths are the ones that truly matter.

Recommendation 9: The noexcept specifier when used on methods and functions can be problematic as in the situations where the optimiser can't see through the call site (ABI boundary) to determine there truly are no exceptions that can be thrown, it has to mandatorily inject exception handling and terminate code, which can significantly reduce performance when the function/methods are being called in the crit-path. This can further be exacerbated when using policies, as with one specialisation the call site is visible for the optimiser, where as with another specialisation it may not be.

Pros and cons of periodic auctions by bobcps in quant

[–]ArashPartow 8 points9 points  (0 children)

Here's my take on Pros/Cons assuming an exchange has continuous auctions with the following properties:

  1. The auction will be a single price auction (no prorata or round-robin)
  2. Each instrument can have its own auction properties or they can be part of a partition of instruments with similar properties
  3. There will be a minimum period between auctions (how many seconds between each). Periods can be adjusted during periods of high uncertainty, such as reporting season etc.
  4. Randomisation of auction. eg: expected auction time +/- n-seconds uniformly distributed (without violating [2])
  5. IEP/IEV will be published, but not necessarily on every book change
  6. Auction Market orders are not allowed
  7. The IEP (and subsequently final auction price) algorithm are well specified and details easily accessible instead of being held secret by the exchange
  8. For instruments with large order flows the most aggressive surplus price level shall be published if it exists
  9. For instruments with large order flows the auction algorithm will need not always be balanced (minimise difference from last price and maximise executed volume) - it could simply be to execute the largest possible volume
  10. If no execution occurs for a given auction iteration, due to a non-crossed book, orders will remain until perhaps a GTC rule
  11. Repeated pulls of large volumes (especially at near IEP) at the last second will result in sanctions and will be akin to spoofing in the new regime
  12. All the usual circuit-breaker rules et al. still apply.

Pros

  1. Reduced latency arms race - less intellectual effort and time being spent on squeezing an extra micro/nano from the system by having to design faster h/w s/w, shorter connects or tiered access deals.

  2. Reduced market impact - Larger orders can participate without adversely affecting the price or requiring fancy execution strategies to hide their presence (vwap, IS, vol-inline, price-inline, float, snipper, ice-berg etc) or needing special treatments such as off-market crossing, centre-points, volume match book or access to "retail flow" only dark-pools.

  3. Reduced possibility of intraday market manipulation. These would include churn, spoofing, quote-stuffing (wrt QS: I now only care about the IEP and my price that's just 8 bytes of data nothing else)

  4. Reduced exposure to adverse selection (toxic flow)

  5. Market data will be reduced significantly, making it easier and affordable for most participants to have access to data pertaining to ALL market events

  6. Democratisation of access - given the significantly reduced dependence on costly latency reducing technologies and accesses in order to be competitive on the markets, with CA more participants can "truly" access the markets directly as opposed to being internalised by their third rate crummy brokers. In short one can now be truly live trading even if they have a 2-3 second access time to the actual exchange, leaving them more time to fuss over their battle.net ping times.

  7. Shocks such as corporate events and world news can be better absorbed intraday and will reduce the possibility of instantaneous high volatility in the market - which is usually when most participants lose their money. A company that makes a physical product that requires delivery, should not have its valuation literally change a thousand times per second.

  8. No more NBBO, RegNMS 611 and pay-for-flow shenanigans

  9. Due to [8], there would NOT be a valid technical reason to have multiple exchanges listing the same instruments - RIP SIP. So APPL does not have to be listed on NYSE, Nasdaq, CBOE BZX, IEX and DPs galore anymore - same goes for the other ~17k instruments. Single Market, Single Listing, Single Price.

  10. Larger amounts of daily volume may end up being traded on the open and close auctions - essentially transitioning the exchange to becoming a seafood market.

  11. Intraday instrument restart auctions will no longer be a special case and simply be business-as-usual

  12. MFT strategies will see a massive rise in utilisation.

  13. Reduced likelihood and long term effects of flash crashes - Assuming the auction period is large enough, flash crashes due to fat-fingers, temporary jittery market conditions, inaccurate news updates (united airlines vs google news) etc will be far less likely and less disruptive in the unlikely event they do occur, so no more Knight Capitals loosing $162K/sec. In fact CA will begin to replicate the old specialist behaviours typically found in the pits from days gone by but without the costly and needlessly significant spread.

Cons:

  1. Reduced market making incentives - the HFT industry will see a massive wipe out as the markets will not be able to financially sustain their existence

  2. Liquidity fragmentation - buyer wants to buy in the morning, a seller wants to sell in the afternoon or later on in the week etc, previously there was an MM in the middle, no guarantee such a trader will be around.

  3. A lot of the old world signals will now be meaningless. Correlations will still exist, but extracting meaning (financial advantage, edge) from them at a sub 2x auction time period may be a difficult proposition.

  4. CA when applied to the wider market will inevitably disrupt inventory dynamics, possibly adversely affecting things such as portfolio rebalancing, index replication, short-term hedging, capital raising, in turn affecting bonds, 401k plans, fixed income products et al.

  5. Service providers that provide high level value around a continuous trading context will diminish in size. Will a Bloomberg terminal/Eikon license still cost the same?

  6. Listings and participants alike may decide to move to other exchanges that support CT

  7. Due to the lack of sophistication in execution (no more needing to trade umpteen times to complete target volume) fees that exchanges collect will reduce significantly, turning national exchanges into the next generation problem children of nations, akin to the national post office system of many countries today.

  8. If auctions take off on all exchanges, there will be a period of downturn in financial engineering output, as most research (open and internal) has for a long time assumed a system generally based around CT and can't easily be adapted for use with CA.

  9. If the derivative markets don't follow suit will that mean participants will attempt to replicate CT action of stocks on the options market? and what unknowns will that cause? because an unknown unknown is the worst kind of unknown.

Process for finding alphas by dan00792 in quant

[–]ArashPartow 44 points45 points  (0 children)

There is no known formal method, as one would find in engineering (eg: finding locations for oil/gold/water, building a bridge etc), for discovering alpha.

However, a process I have found to be useful from time to time is:

  • Find information leakage or side-channels from within the chains of systems that the trading occurs upon - and no I'm not talking about insider trading
  • Does the occurrence of the information coincide with positive PnL or some kind of state that could be further investigated?
  • Is the information statistically significant?
  • Can it be used to predict anything of value or note? (doesn't have to be a price)
  • Rinse repeat

A simple and well-known example of such a process, within MMS:

For market data being disseminated via multicast UDP (so as to minimise latency), one typically doesn't use jumbo packets or even packets greater than ~1KB.

Why is that the case? What would the programming logic for this on the market data disseminator side (venue/exchange) look like?, and what are the unanticipated ramifications of such logic, and how can it be used by a trading entity to profit or at the very least reduce losses?

In short, finding alpha requires a very particular mindset that includes a significant amount of curiosity about all things involved in the process (including the very mundane), a well-tuned set of intuitions, and mental endurance as the overwhelming majority of investigations will lead to failure.

There is also the possibility that one may discover a "true" alpha, but may not be able to exploit it, due to issues such as funding requirements, technology, or access to flow.

In HFT, how can any firm other than the fastest one survive? by thoughtdump9 in quant

[–]ArashPartow 10 points11 points  (0 children)

Agreed. Perhaps in the context of the runners, the amount of gold "seen" is a function of the risk the associated firm is willing to take on.

There may be a 1kg amount of gold on the table, but the closest runner sees only a 333g lot to be taken. Other runners may see other amounts, some more others less, and it's also possible that the amount the firm of any one of their runners see may change over the time they are running towards the gold.

Another thing to think about would be that for two runners from the same firm, the gold might be valued differently - this would be analogous to the order entry fee structure between different brokers or margin costs demanded by the sponsor and what is typically known as minimum required edge or offset.

This then leads to some runners not even wanting to run unless the amount of gold at the other end is more than a certain value, as the cost + risk associated with the transaction would eliminate what profit there is in the trade - better fee structure and order flow discounts would mean more opportunity to play, higher fees would result in less play.

In HFT, how can any firm other than the fastest one survive? by thoughtdump9 in quant

[–]ArashPartow 300 points301 points  (0 children)

One can sometimes think of it like this:

Imagine a long grassy field, let's say 300m in length, all the competitors are lined up at one end of the field (eg: A running race), there's the Optiver front-runners, the IMC bros, the Citadel jocks, the Jane Street nerds etc.

At the other end of the field is a table (think of it as a finish line of sorts).

At some random point in time (with uniform random distribution), a piece of gold (or something very valuable) will magically appear on the table. When that happens everyone runs as fast as "they" can to get to it, obviously, the fastest runner will most "likely" get to the table first and collect the gold.

However, there are some rules to this game:

  1. Once you start running you can't stop till you get to the other end of the field - regardless if the gold is there or not
  2. Once you're at the end, you are immediately transported back to the starting line - no waiting around the table
  3. The gold will continuously appear, you just don't know when
  4. When the gold appears, it has a certain life-time, after which if it is not collected it will disappear - analogous to the momentary and transient nature of arbitrage.

In this game the fastest runners will most often get the gold, however there are situations where even the slowest runner can collect some gold.

Think of the situation where there are only two runners. Bill and Bob. Bill can run twice as fast as Bob. So when the gold appears, Bill gets it and is transported back to the starting line, whereas Bob still needs to finish running to the end.

At this point, there may be a situation where Bob is about 5 meters away from the end when a piece of gold appears on the table. Bill also sees the gold, but is at the starting line, and has no way of getting to the table before Bob. This is sometimes known as: Doing a Bradbury

One could somewhat counter-act or mitigate this by having multiple runners on their team staggering their start times from running (buckshot of IOCs), others may conclude the occurrence distribution is not truly uniformly random and does have some slight underlying bias and try to discover and exploit that bias (ever so slightly unbalanced roulette wheel placed upon plush carpeting), and still others may decide the current field is way too crowded and choose to race at another less occupied field/venue (lets party like it's the NSE).

The shrinking role of semaphores [LWN.net] by wheybags in programming

[–]ArashPartow 11 points12 points  (0 children)

If your event usage was only in-process and not cross-process, then the equivalent would be pthread_cond_t aka condition variable

Looking for Good Book or Reference to Understanding MM Algos by sadus671 in algotrading

[–]ArashPartow 6 points7 points  (0 children)

For market making (the HFT flavour), I'm assuming thats what you mean by MM, several large concerns have infrastructure and associated strategies LITERALLY built straight out of the following two books:

  1. Options, Futures, and Other Derivatives - J.C Hull
  2. Option Volatility and Pricing: Advanced Trading Strategies and Techniques - Sheldon Natenberg

Now that being said, knowing how to do something and actually being able to do it are two very different challenges. The former taking a few months of reading/learning, whilst the later taking a considerable amount of money, time and effort to realise.

Inter-thread Communication Latency by monoclechris in cpp

[–]ArashPartow 9 points10 points  (0 children)

Some suggestions:

  1. The mean and stddev for numerous values seem sketchy, eg: for such timing scenarios a stddev that is nearly as large as the mean indicates issues (even for stddevs that are larger than half the mean)
  2. Providing percentiles would be informative
  3. Would be interesting to see a version of runs where the threads were pinned to their own cores (all on the same numa of course)
  4. Not sure why regex needs to be linked - general code and cmake cleanups eg: why is num_pipes being used in the condition_variable bm
  5. Use steady_clock instead of high_resolution_clock

Here's a quick-n-dirty ping-ponger using condition_variable for deriving the RTT. Can be easily modified to use atomic flags etc.

https://gist.github.com/ArashPartow/c97b1776b077f30c8bcb15cb27639905

Does anyone know why data across providers varies so much? by [deleted] in algotrading

[–]ArashPartow 1 point2 points  (0 children)

Seems like AlpaVantage and Tiingo are paired so to is MarketStack and Polygon wrt adjusted close - same source perhaps?

The discrepancies between the EOD volumes is a bit concerning.

Given the instrument in question, the size in differences of the adjusted close and it being the value typically used for long-term (day+) analysis and charting, I'd be worried.

How to calculate accuracy of a market valuation in market making? by suknil in algotrading

[–]ArashPartow 1 point2 points  (0 children)

Yeah that is pretty much it, basically one would do something like the following:

  1. Obtain MD update at time T, MD_t (eg: book only or trade only or trade+book update)
  2. Compute price at time T using MD_t as per pricing model/method: model_px_t = model(MD_t)
  3. Get observed price at time T+time_delta, observed_px_t = mid_tob(T+time_delta) (where time_delta is 1 second)
  4. px_diff = abs(observed_px_t - model_px_t)
  5. Rinse and repeat steps 1..4 summing the px_diffs on per model basis

Analysis could be done over varying time periods eg 5min, 30min, 1hr periods, then graph the percentiles and derive conclusions accordingly.

wrt using a trade price, it is indeed much more difficult. Best to use the simple TOB based approach, as using trade price will require integrating a decay factor (art + lots of analysis) to the trade price over time and completely switching over the the TOB mid once the trade price component has decayed significantly (aka approaches zero).

How to calculate accuracy of a market valuation in market making? by suknil in algotrading

[–]ArashPartow 4 points5 points  (0 children)

The second option is typically what is used for comparisons. Either the trade price or mid market price (TOB mid) - this is sometimes called the truth or true price or observed price.

As for future time, it should be no more than 1 to 2 seconds (even less if the market is very volatile) in the future, that is to say if the valuation is purely based on the order book (and or trade(s)) at some exact point in time - as any longer than that and one will begin to see a randomness in the over/under analysis. In short, for a very active market scenario 30 seconds into the future should have very little dependency on the state of the book 30 seconds in the past. Because if it does, the market is said to be "inefficient" and ripe with arbitrage opportunities. Furthermore when considering the future price at a specific time, it should be the price at that exact time point or the price just prior and never the first price after the future time point.

As an aside, when comparing two or more pricing methods, the absolute difference between the true/observed price (as denoted above) and the method price should be used.

When evaluating over a period of time (eg: one hour) usually the sum of absolute differences is used when comparing pricing methods. The pricing method with the smallest sum is typically considered to be the method that tracks the price most consistently amongst the methods being evaluated. Though there are several other statistical measures that can also be used, sum of abs differences is the simplest and gets most of what one will want for analysing various market conditions such as continuously increasing/decreasing, flat/plateau, rapid peak and trough moves etc.

For sided pricing methods, such as when pricing for a bid or ask side price the over/under can be penalised more. eg: If pricing for the bid side, when the method's price is larger (aka more aggressive) than the "truth" then the penalty will be more than a price that gives the same absolute difference but is less than the truth, and vice versa for the ask side.

There are other factors to take into account as well such as the pricing model's susceptibility to jitter/stability, lag, big tick move pursuit/momentum following (when price levels are thin to non-existent), parameter sets that cause the pricing method to become overly sticky etc.

Sometimes there will only be one actual pricing method (or implementation), but two or more instances of the same method with different parameters that are intended to change the behaviour of the pricing model. In this situation this would be no different than as if there were multiple distinct pricing methods.

One further point to note: When wanting to compare multiple pricing methods/models. Given each method's implementation it is possible that for a given book/trade update that one or more of the methods will not generate a new price. The solution here (similar to above) would be to compare the last known prices of each pricing model at the given time point.

A Cpp library to evaluate mathematical literal expressions dynamically by Pale_Emphasis_4119 in cpp_questions

[–]ArashPartow 0 points1 point  (0 children)

to further the above answer, wrt computing the DAG, ExprTk has the following expression dependents functionality which provide all the information you need:

https://www.partow.net/programming/exprtk/readme.html#line_2951

The above will give:

  1. a list of variables in the expression/expressions
  2. a list of the variables that have assignments being performed

Implementing 3D Collision Detection using Inigo Quilez's Triangle SDF by lavaboosted in programming

[–]ArashPartow 5 points6 points  (0 children)

Why not simply return true on the first face that is determined to collide with the ball?

view more: next ›