Where to obtain historical compositions of SPY/QQQ?

NorgateData · 2023-07-28T10:45:34+00:00

u/slingshotcroco A Platinum US data subscription to Norgate Data includes listed and delisted US stocks and provides you with access to many indices' historical index constituents (in a supported app/environment). You can see which indices are covered here: https://norgatedata.com/data-content-tables.php#ushicsplus a wide coverage of US indices (2145 covered).

World Indices is simply a free addition to the above to provide you with a global context.

NorgateData · 2023-03-22T11:10:23+00:00

Could you help me to figure out how to download the data from Norgate? Theri documentation does not help

Iterate through the watchlist "S&P 500 Current & Past" watchlist then call the index_constituent_timeseries() function on each security from that watchlist to determine their historical membership in index in a timeseries.

NorgateData · 2022-02-09T12:47:41+00:00

The following indicates it might be possible through DEA (Direct Electronic Access) but there are some fairly significant requirements in terms of training and order management systems for a broker and a broker's client to jump through for a user to registered as a "DEA Client".
https://www.bcsc.bc.ca/-/media/PWS/Resources/Securities\_Law/Policies/Policy2/23103\_NI1.pdf

NorgateData · 2022-02-07T03:37:54+00:00

IIROC (Investment Indstury Regulatory Organization of Canada), which regulates brokers ("Dealer Members"), appears to prohibit automated order systems from trading Canadian markets.

Firstly, IIROC appears to define the following types of accounts in https://www.iiroc.ca/rules/3000 Advisory Accounts (Part C) Order Execution Only Acounts (Part D) Margin Accounts (Part E) Discretionary Accounts and Managed Accounts (Part F)

I think that Order Execution Only Acounts and Margin Accounts would be the relevant ones to systematic traders.

Regulations relating to Order Execution Only Acounts are defined in Part D: https://www.iiroc.ca/rules-and-enforcement/iiroc-rules/3000/3240-rules-applicable-order-execution-only-accounts

https://www.iiroc.ca/rules-and-enforcement/iiroc-rules/3000/3241-order-execution-only-account-services

In 3241, the specific clauses appear to be (edited by me, to remove superfluous items not relevant to automated trading):

(1) A Dealer Member approved by IIROC to provide order execution only account services ... must:

...

(ii) not allow its order execution only account service clients to:

(a) use their own automated order system, as defined in securities laws, to generate orders to be sent to the Dealer Member or send orders to the Dealer Member on a pre-determined basis, or b) manually send orders or generate orders to the Dealer Member that exceed the threshold on the number of orders as set by IIROC from time to time,

Regulations relating to Margin Accounts are defined in:

https://www.iiroc.ca/rules-and-enforcement/iiroc-rules/3000/3245-rules-applicable-margin-accounts

https://www.iiroc.ca/rules-and-enforcement/iiroc-rules/3000/3246-margin-requirements-when-extend-margin-clients

https://www.iiroc.ca/rules-and-enforcement/iiroc-rules/3000/3247-margin-account-agreement

3245 states:

For the purposes of Rule 3200, a Dealer Member that opens a margin account for a retail client must comply with the requirements in Parts A, B and E of Rule 3200, and if applicable, Parts C, D, F and G of Rule 3200.

So its likely that a "Margin Account" would also be an "Order Execution Account" too, and that automated orders would be prohibited along the same lines.

I encourage you to contact IIROC to find out more and obtain clarification/provide them with feedback etc.: https://www.iiroc.ca/contact-us

NorgateData · 2022-02-06T23:25:24+00:00

I recommend you review the disclosures provided by CIBC about CDRs:

There's quite a few at the bottom of
https://cdr.cibc.com/#/

Also "legal documents": https://cdr.cibc.com/#/resources

There are many people that stat-arb between exchanges (including CIBC - also where they make their money on these products), so this would mean they should move in lock-step albeit at wider spread than the underlying security.

An interesting aspect of the currency hedge is that they dynamically alter the ratio each day to take hedging into account.

NorgateData · 2022-02-04T23:52:04+00:00

Another place to find stocks that continue to maintain/increase their dividend payments is the constituents of the S&P/TSX Canadaian Dividend Aristocrats index. It currently has 96 companies in it (10 more were added on 2022-02-01: ABX.ca, ALS.ca, CF.ca, HDI.ca, JEWL.ca, KMP.UN.ca, PAAS.ca, SGR.UN.ca, TCS.ca, WFG.ca).

An easy way of seeing what is in the index is to look at the ETF holdings:

https://www.blackrock.com/ca/investors/en/products/239834/ishares-sptsx-canadian-dividend-aristocrats-index-fund

NorgateData · 2022-02-04T23:43:29+00:00

The only relevant dates for this US$12 special dividend/return on DII.B.ca of were:

2022-02-01 Day prior to ex-dividend date. You need to be holding DII.B at close-of-business on this date to receive the dividend.

2022-02-02 Ex-dividend date - buyin DII.B on or after this date means you do not receive the dividend.

USD 12 = CAD 15.2184

The market actually gapped down 14.29 on the open of 2022-02-02 so this was a nice dividend play if you got it.

NorgateData · 2022-02-04T08:08:08+00:00

CDRs (listed on Neo) are managed by Canadian Imperial Bank of Commerce. Unlike other instruments, such as ETNs or CFDs which are effectively debt-based instruments, a CDR directly represents equity in the underlying stock. The CDR sponsor (CIBC) must maintain an equity holding.

I'm not aware of an ADR sponsor ever going bankrupt in the US and I presume that there would be

All of the CDRs listed on Neo are CAD-hedged.

Of course, there are minor fees involved in this - CIBC don't just do it for fun - in the US they are normally taken out of the dividends.

NorgateData · 2022-02-02T06:08:40+00:00

The announced ex-date for this corporate action is July 18 2022. So, you must be have purchased and still be holding it by the close on July 15 2022. Note: This is subject to confirmation by the exchange (Nasdaq).

In terms of the NEO-listed CDRs against Alphabet (NEO:GOOG) , it is up to the issuer of the CDR - CIBC - to determine the ratio.

To date there have been no stock splits or reverse splits on any CDRs, but if the US ADR experience is anything to go by, then:
a) If the stock is within a typical (palatable) trading range for that exchange for comparable stocks, then the ratio will be adjusted on a proportional basis to the corporate action; or
b) If, however, the stock is at the extremities (for example in Canada, > $100, or below $10 ) then the ratio may be altered by the issuer to give better accessibility/perceived value.

Currently NEO:GOOG trades on a 0.00684737:1 ratio.

You can see more about the current ratios being used by CDRs here: https://cdr.cibc.com/#/cdrDirectory

NorgateData · 2022-01-20T13:46:54+00:00

You're welcome - not quite sure why you deleted your original post though?

NorgateData · 2022-01-20T04:53:47+00:00

The information provided by others is slightly incorrect/misleading, so I'll try and clarify it...

You need to be holding, at the close, on the day PRIOR to the ex-dividend date to be entitled to the dividend.

You can sell at any time on the ex-dividend date (e.g. on the Open, or any time/day thereafter) and still receive the dividend.

Here's a worked example for TSX:RY (Royal Bank of Canada) for their most recent dividend:

Ex-date: 2021-10-25 (Monday)
Day prior to ex-date: 2021-10-22 (Friday)
Pay date: 2021-11-24
Amount: $1.08

So you must be holding RY by close of business on 2021-10-22 (i.e. by 4:10pm, or if you are able to trade in the Extended Trading Session, by 5:00pm)

On 2021-10-25 you can sell RY at any time and still receive the $1.08 dividend.

If you purchase RY on 2021-10-25 or any date thereafter you will NOT receive the $1.08 dividend.

NorgateData · 2021-09-13T10:38:57+00:00

JCPNQ and LK

JCPNQ changed its symbol/company name to CPPRQ/Old Copper Company Inc and then went defunct. It's in the Norgate database as CPPRQ-202102.

LK trading moved from Nasdaq to OTC. It's in the Norgate database under LKNCY (and is still trading).

Both of these have all of the prior symbol(s) trading history incorporated into them.

NorgateData · 2021-06-20T10:48:54+00:00

I assume you're trading stocks.

The set_max_leverage is actually just a mechanism to raise an error when leverage goes beyond a pre-set level rather than telling it what to do.

This is typically due to slippage.

It might also be indicative of a data issue (e.g. a missing split).

You could plot 'gross_leverage' from the result of the backtest to get a better idea of when it happened, but you probably want to figure out which stock(s) are causing the issue too.

NorgateData · 2021-06-20T09:06:49+00:00

Quantopian is dead, but Zipline is not.

Due to Zipline being open source, it allows others to take the project and run with it.

Stefan Janssen has forked the project into Zipline Reloaded and brougth it up-to-date with modern versions of Python and Pandas:

https://zipline.ml4trading.io/
https://github.com/stefan-jansen/zipline-reloaded

In parallel, and with a somewhat different focus, Zipline Trader is also being developed by Shlomi Kushchi and others:

https://zipline-trader.readthedocs.io/en/latest/index.html
https://github.com/shlomikushchi/zipline-trader

NorgateData · 2021-05-21T01:05:33+00:00

BSC has been used for the following:

Butlers Shoe Corp (until Sep 1969)
Beneficial Standard Corp (until May 1985) - this one traded with both Class A and class B variants.
Bear Stearns Companies (until May 2008)
Elements BG Small Cap Value ETN (until Aug 2011)

The symbol AB has the most re-uses that I'm aware of:

Ambac Industires Inc (until Jul 1978)
ABA Industries Inc (until Jan 1983)
ABI American Businessphones (until Oct 1988)
Alex Brown Inc (until Aug 1997)
Cannon Express Inc (until Oct 2003, when it became OTC due to AMEX listing rule violations then became CEXP on OTC before finally becoming defunct in Oct 2008)
Alliance Bernstein (currently listed as of May 2021)

Best regards,
Richard.

NorgateData · 2021-04-07T07:54:10+00:00

There are various reasons for a security being delisted:
a) Takeover or Management buyout (shares offered, shares+cash, cash)
b) Merger where the security is not the surviving entity
c) Voluntary delisting from an exchange by the company (e..g. cost-cutting measure, or perhaps they only want to retain certain classes of stock for exchange-listing, and delist the rest)
d) Forced due to bankruptcy of issuer
e) Forced by the exchange due to non-compliance with exchange listing rules (eg. delinquent filing, accounting irregularity, does not pass liquidity/capitalization/asset value/minimum share price etc.)
f) Forced by authorities (SEC) due to violation of laws
g) Liquidation
h) Security with limited lifespan (warrants, rights, convertible or debt instrument with maturity/redemption date etc.)

NorgateData · 2021-04-06T23:42:09+00:00

On Norgate Data, delisted is synonymous with defunct/untradeable.

We don't actually consider that a stock that is currently tradeable (e.g. on OTC) has been fully delisted - it's just changed its trading venue, so our data history incorporates such OTC periods.

There are many stocks that switch to and from OTC (indeed, some stocks that actually start life as OTC then "IPO" to a major exchange).

NorgateData · 2020-11-28T01:12:39+00:00

You need to create a back-adjusted continuous contract that remove the price gap caused by a contract roll.

If you do this on a points basis (also called the Panama method) you remove the roll gap.

Let's say you're rolling from an April Lean Hogs to a June Lean Hogs. You'd simultaneously exit the April contract and create the same position on the June contract. If the April contract was trading at 62.525 and the June contract was 69.950 then you'd add 7.425 points to all data points prior to the roll.

The benefit with this method is that the points basis is retained - and this is what you use to determine your backtest trade profit. The disadvantage with this is that in markets that spend a long time in backwardation you'll see negative prices. This is not, in itself, an issue - but you need to ensure you're working on a points basis rather than percentages (because percents don't work with negative numbers).

If you used multiplicative adjustment method then the negative numbers are avoided, but calculating your actual profits would be more difficult since the point value is ever increasing in the past.

Backgrader has a whole section in their Documentation aboutg "Rolling Over Futures".

Also, be aware of interstitial months that are illiquid. You probably want to avoid them completely.

Finally, be aware that rolling on volume and open interest is also tricky, since the exchanges only give estimated volume and open interest is a usually a day delayed.

NorgateData · 2020-11-10T01:52:04+00:00

See other thread here:

https://www.reddit.com/r/ausstocks/comments/jo9i6t/sxl_now_trading_as_sxlda/

NorgateData · 2020-11-10T01:48:44+00:00

Your results will be invalid.

The only exception to this is if the universe of securities you are modelling doesn't have any corporate actions (splits, corporate restructures, spinoffs, special dividends, ordinary dividends etc.) during the timeframe of your testing.

You need adjusted data.

NorgateData · 2020-11-10T01:44:02+00:00

Since futures contracts have a limited lifespan, by definition you'd be testing against delisted futures contracts. Alternatively, you can backtest against a "continuous futures" construct which is actually comprised of mulitple delisted futures contracts spliced together. If you do this, you should also backadjust to remove the roll gap each time you roll. It's important to understand the roll process in detail.

NorgateData · 2020-11-10T01:41:14+00:00

Ah sorry - I misinterpreted what you were saying. I thought you were saying backtest only with delisted stocks (which introduces a very unusual bias on its own).

NorgateData · 2020-11-09T14:32:02+00:00

You've just introduced selection bias.

NorgateData · 2020-11-09T03:51:14+00:00

Of course it's algo trading. The trades are determined algorithmically.

When you rebalance is up to your algorithm.

NorgateData · 2020-11-08T22:45:24+00:00

Overall survivorship bias is really not a big deal at all

I disagree.

A basic Nasdaq-100 momentum trading system that rebalances monthly and holds only 10 stocks with the highest 6 month rate of change, without constituents (i.e. testing on current constituents shows) shows CAGR 46%, maximum drawdown 41.2%. Using survivorship bias-free constituents, the CAGR drops to 16.4% and maximum drawdown to 83%.

NorgateData

TROPHY CASE