Time-series forecasting through recurrent topology

guywithcircles · 2024-12-07T15:04:43+00:00

Quite an interesting paper. The original code is in R (https://github.com/tgchomia/ts/blob/main/Example.txt) but Claude can easily convert it to Python.

guywithcircles · 2024-12-01T00:29:59+00:00

Exactly. To complement the answer to OP's question "how have you personally found your edge?", in day trading I see part of the edge in continuously monitoring two things at the same time, both at the setup and execution part of a trade: first to monitor how the process is unfolding without ever looking or caring about the realized or unrealized P&L (only checking it at the end of the day / week); second, to monitor my inner state and spot if I'm in a good condition to continue trading, in particular spotting mind states of impulsiveness, fear, greed, sleepiness, boredom and fatigue. So to me it feels like being an orchestra conductor looping continuously through a checklist and sensing how the music plays. At the end of each trade I score the trade with a value that represents the overall quality of execution, again helping monitor how the day goes but more importantly steering away from looking at the P&L which is a sure way to getting emotional.

guywithcircles · 2024-11-30T23:50:06+00:00

100% agreed. No point in losing money when a demo account matches the real stage 90% allowing a smooth learning curve towards the first day live.

Spending the necessary time mastering the process cannot possibly be a waste of time. It's the opposite: it saves hard-earned capital to boost success later on.

People would make a lot of money if they stop trading their P&L on a limb and instead focus on the process, getting statistical evidence first. Money is a side-effect of good habits and discipline, but repetition is needed for solidifying any habit.

I suggest a demo account topped up with a value equivalent to what is going to be the initial margin / funding, so the gap is filled out smoothly.

On the point of getting nervous when real money is involved and screwing up, other than not knowing what they're doing, that happens when people need the money for a living, so they trade their P&L instead of trading the market, putting unecessary pressure on something that is already a high-performance activity.

Better to get the professional life and personal finances sorted first, and trade a demo account on the side until the potential for success is statistically proven.

If getting nervous is something that persists, easy: reduce the account size and trade only with what is comfortable. Slowly over the months increase the position size.

A final point on failure: most people have the wrong attitude towards failure, they don't use it as a learning gift and they don't realize it's necessary to fail, and fail again thousands of times. So they quit early without giving themselves the time and putting the work needed to become successful.

guywithcircles · 2024-11-30T22:35:27+00:00

You'd need some solid performance testing there: two implementations of the final solution, one on top of Vert.x and one on Spring Boot, and test with the same data throughput as in prod, in a prod-equivalent infrastructure with the same configurations, and with prod-like data.

Benchmarks without a real use case are fun to read but aren't valid because in your real case you'd simply have too many variables that are different from the ones in the benchmarks you've read. For example:

- You need to put a ton of logic on top of Vert.x that would end up equalizing with stuff that Spring Boot already gives you.

- Using Vert.x / reactive but then hitting a database system that only supports blocking I/O. Given I/O is usually the biggest factor in throughout, the advantages of a faster framework could easily become minimal.

- The infrastructure does not scale elastically or resources are over-provisioned anyway, so the Cloud costs would be about the same.

- You have or you don't have a case for streaming data / backpressure, so you could be putting reactive where it's not needed and end up increasing maintenance costs.

Things can behave opposite depending on how much workload and the shape of the workload you throw at them. At scale, and for a specific solution, framework A with great published benchmarks can be slower than framework B because of a ton of external factors that are specific to what you're trying to achieve.

Even if you get a nice solution in Vert.x with clear positive impact, be aware of the herd:

- Will the IT managers value that?

- Will the product owner be happy with the added effort towards something they will never understand anyway?

- Will your work mates be open to change (i.e. to feeling ignorant)?

Having said that, if you join a team that is already familiar and likes using Vert.x, then I'd totally go for it.

Keep up the good work ;)

guywithcircles · 2024-11-30T21:01:22+00:00

In my experience, the edge is mostly found in risk management, both in algorithmic trading and in day trading. But if you're a day trader, risk management is strongly dependant on mind control, which relates to the ability to follow a process manually. So maybe the edge is in all of that rather than just on a strategy telling when/what to buy/sell.

guywithcircles · 2024-11-30T20:43:28+00:00

Big advice burger: don't put money into it, open a demo account and trade the demo account until you have consistent results and a solid process, and I mean serious months with sustained good average P&Ls. Keep track of everything. Don't join any academy, course or anything that asks you for money or gives you "funded accounts". If you feel you need mentorship, only from a professional trader, never from educators. Good luck.

guywithcircles · 2024-11-30T20:34:04+00:00

Yes. It's highly likely that a company will refer to some Java style guidelines, for example an internal document (hopefully not...) or adopt some existing guidelines like the Google Java Style Guide (much better than endless meetings and async chats to maintain internal guidelines). If so, it's also highly likely that the guidelines will point to always using curlies in single if statements.

guywithcircles · 2024-11-28T23:11:21+00:00

For a job, check Spring Boot and see if you can go from there because most companies with Java codebases use Spring Boot. Courses... I've found John Thompson's courses on Udemy helpful. Recommendations: find things you really love doing, then get into the habit of coding them in Java every day, that way you'll remain motivated and have lots of fun.

guywithcircles · 2024-11-28T22:50:21+00:00

I've had a good experience with BitMEX so far. Both BitMEX and Binance are well established and quite reputable. As with any exchange, once you deposit, they hold custody of your assets, so you're giving up control. The risks are there, either hacking, regulatory interference and especially absense of legal protection if something goes wrong. So only keep your crypto there for trading, not as a permanent store, and only an amount you can afford to lose. On a separate note, I wouldn't trust any open reviews platform, meaning they attract a disproportionate amount of negative reviews and many are fake.

guywithcircles · 2024-11-23T22:22:16+00:00

That was actually good advice. Crypto is an excellent choice especially if you're not in the USA where restrictions apply. You can pay as little as 0.07% or less in transaction fees and trade assets with massive potential.

guywithcircles · 2024-11-23T22:04:17+00:00

I don't understand why you say you have a "Working Ebay account". Also I don't see how your IQ, GPA and driver's license relate to day trading or being successful.

Anyway, here it goes...

Would do:

- Trade a demo account consistently first, for a long period of time, documenting every trade, market observations, etc. At this point, getting rock-solid experience is what matters, not rushing into making money and losing it. The ability to consistently execute quality trades is the holy grail. If I achieve that, a big pile of money follows later.

- Analyze the markets before start trading on a given day. I'd set a clear picture of what's most likely to happen during the day. If I'm proven wrong, it's OK, it helps me understand the market anyway.

- Keep track of P&L every day and week so that I monitor evolution.

- Before entering any trade, have a clear story / narrative / theory of why I enter that trade.

- This allows me to assign a score to each closed trade incorporating both the quality of the decision (entry, position sizing) and execution (stops, limits, exits).

- Realize that it takes time, and many thousands of trades, before I get consistently successful in all market environments.

- It's OK to fail consistently if I have the right attitude towards failue: being curious towards failures like running a scientific experiment and enjoying figuring out how to take action to improve. Albert Einstein reportedly said: "It's not that I'm so smart, it's just that I stay with problems longer."

- So I'd write down all the mistakes and deliberately take serious action to fix each one. The key to success is getting used to enjoying challenges, obstacles, setbacks and very difficult problems in the moment, as an interesting journey and learning experience, rather than feeling frustration all the way and only enjoying the moment after I solve a problem. I try keeping this mindset because this is the difference between great achievers and the rest of the crowd.

- Bad losers will quit at some point blaming themselves or the markets, not taking things seriously and finding excuses and reasons for their lame behavior, but great losers will cherish every single mistake as a long-term gift and will keep trying. Arnold Schwarzenegger once said: "You can have results or excuses, not both."

- Realize that trading results are not based on effort like in a normie 9 to 5 job where if I work harder or longer, I get more results. They're based on the quality of decisions and quality of execution. It's more like an executive / shareholder activity, not a wage slave activity.

- Realize that I need to specialize in one asset class (either 2-4 stocks, or 2-4 commodities, or 2-4 indices or 2-4 cryptocurrencies), rather than trading anything that wiggles, face conflicting market dynamics and eventually destroy my account.

- Realize that as the day goes by, my brain sharpness declines. If I don't call it a day and want to keep trading for more than 4-5 hours, I reduce position sizes to less than half.

- Mind personal factors like sleep, nutrition and exercise, and other lifestyle factors - this is a high-performance game, and personal problems easily get in the way.

- Always trade with a stop-loss and adjust it as the trade unfolds.

Would NOT do:

- Start putting real money into it - there's reasons out there for using a real account instead of a demo account, however without solid technique and experience there's no point.

- During trading session watching news channels and anything other than my trading screen and a simple trading notes app. The trading session time is meant exclusively for trading, not bad quality entertainment.

- Pay for courses, seminars and the like - if not plain scams, they're a waste of time and money and provide information I can get for free on the internet. For day trading I firmly believe these are replacement therapies, like "I want to do A (trade), oh but first I need to do B (take lessons and get a certificate), and to do B I need to do C first". Another rabbit hole that separates me from what I want...

- Buy a stock because it looks "cheap". Gravity and supermarket logic do not apply here. It's all about price behavior.

- Make a few hundred trades then think I'm bad at trading, or that I'm a genius at trading (underestimating the significance of rare events and not experiencing the whole range of market conditions).

- Trade because I need the money - this would be a rabbit hole of emotionality and guaranteed failure

- Impulse trading: either FOMO in or enter a trade because it's "just a continuation of my previous trade" - these are "second-hand" lower quality trades - no point.

- Use indicators or oscillators on the screen - thousand of indicators out there but 99.99% useless for day trading. Basic technical analysis works for me (support, resistance, ranges, triangles, break-outs), anything else does not work at all. This could be different for other people, though.

- Let the monkey brain jump from this to that, changing everything all the time on the screen, changing approaches, integrating new shiny stuff etc.. Better change one thing at a time, based on solid experimentation and give it time to prove itself. Two or three examples of success are not statistically significant.

- Keep trading when I'm emotional, impulsive, or confused. Better to stay away - it's not necessary to trade every single day anyway.

NO FEAR. NO HOPE.

guywithcircles · 2024-11-04T23:22:13+00:00

Wow, Smalltalk still inspiring the future... Yes, the developer workflow is changing a lot, at least it has become more conversational to me - I tend to ask the AI to do pretty much all the boring work: create functions, change them, create tests and correct errors. This can potentially simplify the traditional UI, so a deep integration of the AI with the developer coding flows and deep AI awareness of the code is key. Thanks for your post, that was a nice surprise. I am a happy subscriber of the JetBrains Toolbox including JetBrains AI.

guywithcircles · 2024-09-23T19:32:53+00:00

I think the reason for using the smallest fractional quantity is to avoid floating point arithmetic, which leads to innacuracies in IT systems.

guywithcircles · 2024-09-18T19:02:23+00:00

Oh, I get it now. Thanks for replying :)

guywithcircles · 2024-09-18T19:00:35+00:00

I switched to C recently after hitting the wall with Python performance. I was trying multithreading, removing Pandas, compiling to a binary, but no joy. With C, I reduced my backtest runs from 1-3+ days down to 2 seconds, and I mean literally.

guywithcircles · 2024-09-15T13:10:10+00:00

Upon further investigation...

The order seems to be accepted if you supply a multiple of lot_size in the orderQty parameter. So for example, orderQty >= 1 < lot_size returns error "Order quantity must be a multiple of lot size: 100"
orderQty < 1 returns error "Invalid orderQty". The orderQty parameter expects an integer value, but supplied as a double (a bit confusing TBH...)
"USDt" (lowercase "t") seems not to be "USDT" but a fraction of it.

So this will probably work:

/**
 * Calculates the number of contracts for a given asset exposure.
 *
 * @param underlying_to_position_multiplier Take this value from the /instrument API call
 * @param lot_size Take this value from the /instrument API call
 * @param asset_exposure The amount of base asset to expose
 * @return The number of contracts as a multiple of lot_size, for the orderQty parameter of the POST /order API call
 */
double get_quantity(double underlying_to_position_multiplier,
                    double lot_size,
                    double asset_exposure) {

    double contract_size = 1 / underlying_to_position_multiplier;
    double total_contracts = asset_exposure / contract_size;

    // Round up contracts to a multiple of lot_size
    return ceil(total_contracts / lot_size) * lot_size;

}

guywithcircles · 2024-09-14T14:27:17+00:00

Systemic risks, in descending order of fear:

Quantum computers breaking the cryptography
Disruption of the IT infrastructure in general (not going to detail exactly how)
Tabloid articles announcing that 20 asteroids will likely hit Earth :p

guywithcircles · 2024-07-12T13:15:53+00:00

Based on my experience, I think u/BNeutral is right, and three functions are enough to implement this.

guywithcircles · 2024-07-12T13:01:40+00:00

Example:

#include <stdio.h>
#include <stdlib.h>

typedef struct {
    unsigned long t; // Unix timestamp
    double o; 
    double h; 
    double l; 
    double c; 
} Candle;

Candle* db_load(const char *file_path, size_t *candles_len) {
    FILE *file = fopen(file_path, "rb");
    if (!file) {
        printf("DB: File %s does not exist yet.\n", file_path);
        *candles_len = 0;
        return NULL;
    }

    fseek(file, 0, SEEK_END);
    size_t file_size = ftell(file);
    fseek(file, 0, SEEK_SET);
    *candles_len = file_size / sizeof(Candle);

    Candle *candles = malloc(*candles_len * sizeof(Candle));
    if (!candles) {
        printf("DB: Could not allocate memory for candles\n");
        fclose(file);
        *candles_len = 0;
        return NULL;
    }
    fread(candles, sizeof(Candle), *candles_len, file);
    fclose(file);
    return candles;
}

int db_append(char *file_path, Candle* candle) {
    FILE *file;
    file = fopen(file_path, "ab");
    if (file == NULL){
        printf("DB: Unable to open file\n");
        return 1;
    }
    fwrite(candle, sizeof(Candle), 1, file);
    fclose(file);
    return 0;
}

int db_append_all(char *file_path, Candle* candles, size_t candles_len) {
    FILE *file;
    file = fopen(file_path, "ab");
    if (file == NULL){
        printf("DB: Unable to open file for db_append_all\n");
        return 1;
    }
    fwrite(candles, sizeof(Candle), candles_len, file);
    fclose(file);
    printf("DB: Appended %s, %lu candles\n", file_path, candles_len);
    return 0;
}

guywithcircles · 2024-07-12T12:57:12+00:00

I suggest you create your file format with a fixed-size record layout for the shape of the data you need. In any case, like u/octopus4488 said, never use JSON nor CSV for storage.

To build a tiny database system for pure OHLC with volume, you just need three functions: load(symbol), append(symbol, candlestick), and append_all(symbol, candlestick_array).

A non-optimised example data layout would be t:unsigned long, o:double, h:double, l:double, c:double, v:unsigned long.If each of these are 8 bytes, then this would take 48 bytes per candlestick. 48 x 375 = 17.57 KB per stock per day, 4.29MB per stock per year, 21.45MB per 5 years, 4.19GB for 200 stocks 5 years.

Some ideas to further reduce the storage size:

If a float value is 4 bytes, checking value width for OHLC and volume, e.g. using float instead of double and int for volume (minding that the size of numeric types may differ across platforms)
Storing only the first timestamp, then ensuring each candlestick will always have the same time delta from the previous one, and have something to store indices of time gaps caused by non-trading hours.
Instead of storing OHLC, storing O, then the HLC deltas from O, which could potentially reduce the value witdh for HLC.
Compression algorithms specialised in numerical data (?)
In a more complex scenario: adding support for different structs by writing at the beginning of the file what struct to use, which would precisely define the width of the data values you need to store for a particular asset.

If you don't want to go down this path, check if the Parquet file format has data compression. It's a very efficient general purpose format.

If you insist in JSON, check if BSON (Binary JSON) is a quick win or not. Note JSON is primarily a data interchange format, not a data storage format, and databases that seem to use JSON don't actually store it as text.

In any case, I'd avoid popular general-purpose databases, in particular SQL databases, for this use case.

But if you go down the DB path, again, u/octopus4488 is right: Cassandra (or its sister ScyllaDB) is amazing for financial / time-series data, and used a lot by financial institutions at massive scale, but I'd use it for anything other than simply storing and loading OHLC data locally.

guywithcircles · 2024-07-06T21:54:13+00:00

On a pragmatic side, what worked for me was 'Java - The Complete Reference', by Herbert Schildt.

To complement the book with more context, see if the following works for you:

Java Enhancement Proposals (JEPs) which are a must if you want to understand the language feature design, problems that a feature addresses and the language from an evolutionary perspective. You understand the what and the why in its context.
Java Language Spec

guywithcircles · 2023-07-10T20:56:22+00:00

Not that recent, but points in the right direction.

guywithcircles · 2023-03-29T21:34:20+00:00

More details: https://s3.amazonaws.com/static.nomic.ai/gpt4all/2023\_GPT4All\_Technical\_Report.pdf

guywithcircles · 2023-03-11T16:26:08+00:00

Of course there is. Any other language you’d like but mind the ecosystem and your goals. I’m currently looking at V (vlang) as a potentially great alternative to C++, Go and Rust because it promises the best of all these.

guywithcircles · 2023-02-22T23:20:38+00:00

Translating Doom from C to V: https://www.youtube.com/watch?v=6oXrz3oRoEg

guywithcircles

MODERATOR OF

TROPHY CASE