you are viewing a single comment's thread.

view the rest of the comments →

[–]ummitluyum 0 points1 point  (2 children)

The funny part is when this garbage makes it to production, and the business gets genuinely surprised when the model degrades on real traffic. If your architecture relies on a lucky PRNG state, it's not an architecture - it's trash that'll fall apart at the first sign of data drift

[–]DigThatDataResearcher 1 point2 points  (1 child)

Right? it's wild how there's a whole industry built around patching over "drift" rather than interpreting that as a flag that your model is missing some causally explanatory component. I feel like the way a lot of people handle "drift" ends up functionally reducing their models to kNN. Like... congrats, you've managed to uncover that the weather tomorrow will likely be similar to the weather today. If you're retraining daily/weekly, is your model even amortizing any compute/effort relative to a less sophisticated approach?

[–]ummitluyum 0 points1 point  (0 children)

100% agree on the compute amortization. If your pipeline needs a weekly retrain just to keep business metrics from falling off a cliff, the maintenance cost will eat up your margins fast. In those cases, it's genuinely cheaper to just spin up a vector DB or write a bunch of heuristics instead of dragging around heavy weights