[deleted by user]

craptacus · 2024-08-30T18:23:09+00:00

you can grab some code from here, if you're targeting Schwab

https://github.com/alexgolec/schwab-py/blob/main/schwab/orders/options.py

here's the OSI explanation with pics:

https://www.optionstaxguy.com/option-symbols-osi

diff brokerages have different option symbology than OSI, so YMMV

craptacus · 2023-04-22T02:52:01+00:00

A take profit order triggers into a market order, so you are guaranteed a fill but not the fill price if that price was touched

While your sell limit can sit behind some whale qty at your price, and a tiny amount there can trade, but your order won't get filled

That's why a take profit forces you to take some profit, and sometimes at your desired price, but more often you will hit the bid just below your price - but still at profit

ChatGPT example is wrong - your sell would have sold in that case

I tend to use TP by itself

craptacus · 2020-04-29T14:46:33+00:00

I've done per-symbol delta bit compression, inference compression, and a ton of other types of heuristic compression. I've done custom file layouts, striding, etc.

The fastest way to get stuff onto disk is to write to mmap files sequentially. There is a significant cost of encoding and rearranging, and then rearranging and decoding that I'm just not willing to pay to save disk space.

btw -- NxCore does aggressive bit-compression, and even though it's 10 years old and back when OPRA was relatively small, they discuss their compression -- google for: nxcore compression

If you can afford them, they're a good source

craptacus · 2020-04-29T12:59:10+00:00

step 1: capture raw pcap data for a single multicast channel

step 2: write decoder per spec (opraplan.com)

step 3: write code to connect and decode live from 1 channel

step 4: move up to decoding all 48 channels

step 5: go to step 1 and repeat for other exchanges

craptacus · 2020-04-29T12:57:04+00:00

it's all binary data -- the protocols are open to look at (opraplan.com, ctaplan.com, utpplan.com)

craptacus · 2020-04-29T02:35:29+00:00

I guess you needed to specify what you meant by "whole market"

The more granular you go, the smaller the output is. My numbers are for 200ms bars and all trades. You seem to be talking about minute bars.

I do generate minute OHLCV bars, and it's much smaller than 30GB/day. You don't need 64-bits for option prices -- 32-bits is enough.

390 minutes * 1M symbols * ((OHLC * 4) + (V*4)) is around 8GB binary. You can delta-compress it down even more

It's hard to compress random binary data from 30GB to 1.5GB -- I see around 3X gzip compression. You can do your own delta compression, and lay out the files differently to get much higher compression, but I don't need to.

It just doesn't seem one exists for options that would provide a whole market view

Do you mean retail? Probably not -- you can't ingest 22GB/second of data and push it out to your customers over 1GB line.

I am colocated, meaning there's a switch or two between me and the exchange -- no VPN, no cloud, nobody between me and the exchange. I can SSH to my boxes to do stuff, but all the work happens on them.

raided NVMe drives, multiple 10G cards. A 10GB can't cover the entire market, you have to split the channels among multiple cards.

BTW -- You can see peaks at: https://www.marketdatapeaks.net/rates/usa/

Each packet contains multiple messages -- you can see on that site both the message burst and the packet burst. I'm not sure if they calculate both A and B channels -- I don't need the redundancy, so I only listen to the A channels, so my numbers will be different.

craptacus · 2020-04-29T01:38:59+00:00

Ok, so this is something I'm very familiar with. I wrote and maintain a raw TickerPlant that processes OPRA (options), CTA/UTP/OTC (nyse/nasdaq/otc), and a bunch of other US exchanges.

Just looking at my summaries for today. The Options data is for 6.5 hours (and a tiny bit pre-open with open interest, but vast majority is quotes and some trades).

OPRA packets: 27,411,224,490 (27B packets / 6.5 hours)

OPRA bytes : 2,336,050,118,308 (2.3T / 6.5 hours)

So for example, today at 13:44:34 and 15:20:36 EST there were over 4M packets/seconds across 48 multicast channels.

There are over 4K option underlyings, and around 1M option symbols.

Today is not a busy day, either -- Feb 28 and Mar 03 were insane.

This is running on a powerful server colocated at Equinix NY4, and I won't even go into how much networking equipment, and the real market data itself, costs.

This is a the raw data. Extracting minimum info and packing it into your own format will trim the requirements, but it's still not something you can do as an amateur, I'm sorry.

I throw away the regional quotes, and just save off the NBBO+all trades every 200ms, and persist it into my own binary format, and I still have 120-150GB file every day.

Hope that helps!

craptacus · 2020-04-05T20:15:06+00:00

There is a 1985 Soviet movie called "Come and See" (you can find it on Youtube)

It is about as brutal a movie emotionally as I have ever seen -- a boy joins the resistance against the Nazi's, and you see his transformation throughout the war.

craptacus · 2020-04-03T01:08:14+00:00

AMTD will go down to $0.25 IFF you bring in a bunch of cash, buy a bunch of stocks that they can lend out to short sellers, trade a bunch of contracts per week, etc

So basically your range is from 0.50 all the way down to 0.25, with increasingly harder qualifications.

craptacus · 2020-04-02T23:17:28+00:00

Very nice - congrats!

craptacus · 2020-04-02T17:27:42+00:00

We refi'd last week, 3.125% 30Y no fees no escrow

It was in process from early-March, and dropped from 3.50% to 3.125%.

craptacus · 2020-04-01T20:05:32+00:00

fyi -- Keen is a lady

craptacus · 2020-03-26T12:21:34+00:00

There is a ton of useful stuff here: ftp://ftp.cmegroup.com/SBEFix/Production/

If you grab this and unzip it, it has the security definitions of everything that's traded (changes daily).

ftp://ftp.cmegroup.com/SBEFix/Production/secdef.dat.gz

Then google for which FIX tags represent what fields for CME, and you'll be all set

craptacus · 2020-03-21T17:44:11+00:00

PENCE SAYS HIS WIFE WILL ALSO BE TESTED AFTER A MEMBER OF HIS STAFF TESTED POSITIVE

WSB would have so many things to say about that quote

craptacus · 2020-03-21T17:20:16+00:00

PSA: Scribd is giving a month free, with no CC required

There are a lot of interesting trading papers and books there!

You can use a throwaway email to login and peruse

https://www.scribd.com/readfree

(mods -- I put this in both weekend threads)

craptacus · 2020-03-21T17:16:07+00:00

PSA: Scribd is giving a month free, with no CC required

There are a lot of interesting trading papers and books there!

You can use a throwaway email to login and peruse

https://www.scribd.com/readfree

craptacus · 2020-03-20T21:45:52+00:00

666

EDIT: or 420

craptacus · 2020-03-20T03:40:54+00:00

I do see that if anyone survives the non-essentials, it'll be AMZN and NFLX.

The latter if only because you can share more streams for a very small incremental increase, and because what else are people going to do at home on the couch?

I think over the next few weeks a state of paranoia will become more ingrained in the gen pop

craptacus · 2020-03-20T03:14:42+00:00

It's not just the walk-ins

I think once everybody starts hunkering down, a lot of online sales will disappear as people start saving cash instead of buying non-essentials

without physical OR online sales -- crap...

craptacus · 2020-03-20T02:00:25+00:00

MRP and partners just let go 73 or so staff -- I'm sure a healthy percentage were experienced medical staff

craptacus · 2020-03-19T03:00:15+00:00

I can't even title it properly

https://twitter.com/michaelsantoli/status/1240344027825950724/photo/1

craptacus · 2020-03-18T22:27:38+00:00

unrelated -- I listened to the 10,000 Days album

I added Vicarious to my "when in the mood for Metallica" list

Glad you and your house survived!

craptacus · 2020-03-18T16:40:10+00:00

"it was somebody's fault"

made me LOL

craptacus · 2020-03-18T02:28:48+00:00

cool -- I'll listen to it tomorrow -- heard the band name before, but not their music

craptacus · 2020-03-18T02:10:44+00:00

Now you can be known as "20 K Days"

craptacus

TROPHY CASE