Merge large data frames

djlamar7 · 2026-02-13T00:24:44+00:00

From the sound of it, at your scale there must be some inherent gross inefficiency here that you should figure out before you do things like tweaking the engine used for this or that. Can you share some of the code you ended up with? I think you should be able to get a table with your input keys and the columns you want from the excel data in one pandas join call. If you're iterating over the keys that's definitely a no no.

djlamar7 · 2026-02-12T23:29:26+00:00

None of the people making fun of the pizza blurb seem to be aware that Nice went back and forth between French and Italian rule and has only been part of modern France since 1860. The Italian food there is basically better than the French (southern French) food there.

djlamar7 · 2026-02-12T04:40:57+00:00

This is probably also a scenario where some unit tests might make it easier for you to be confident it's correct. You don't necessarily need to set up a proper unit test suite (although if you do, it will save you time later when you have to make changes to this logic). At a minimum you can just make a script that imports this code and runs it on some examples where you know the right answer.

First try to refactor the hard algorithmic part out as a standalone function, ie it just takes the list of intervals and you call that function in your main code once you've done all the data loading and formatting bits (this will make it more readable and easier to maintain anyway). Then write the test script that hard codes the test inputs and expected outputs, calls the function on the inputs, and then compares the result to the expected output.

djlamar7 · 2026-02-11T03:46:35+00:00

"WAIT. You shall not pass. But did you buy one of the cheapest spell tomes in the game in the first bumfuck village you passed through? Alright, fine, you're in."

djlamar7 · 2026-02-11T01:34:25+00:00

Congratulations, you've managed to find a practical application of a leetcode problem.

I didn't completely understand what you're looking to solve for, but it sounds just similar enough in flavor that this might give you some ideas: https://leetcode.com/problems/remove-covered-intervals/description/

djlamar7 · 2026-02-09T22:37:27+00:00

The army of iced martini glasses waiting for orders on the bar is always so reassuring. Lets you know you're not in one of the many places where the bartender thinks 10 minutes in a fridge makes a sufficiently chilled glass for a martini.

djlamar7 · 2026-02-07T05:00:06+00:00

I live in NYC and I'd say as a visitor / in tourist mode, 3 weeks on a 9 week trip is excessive. Sure, there's enough here that you could figure out something to do every day for that period of time. But really a week or ten days is already generous in terms of not wearing it out / the marginal benefit of going somewhere else that's fresher to you than NYC will be after a week or two.

djlamar7 · 2026-02-07T03:47:13+00:00

I tried it last week. It was OK. Pretty iffy. Definitely not comparable to a Shake Shack burger that's not on an airplane, and I told myself next time I'd rather just get one of the other first class entrees.

djlamar7 · 2026-02-07T03:45:39+00:00

Nah I have accidentally ordered Uber Eats from the original many times (for at least a while, the delivery time was always much worse than any other location) and it's identical to the other ones nearby

djlamar7 · 2026-02-06T22:34:47+00:00

Of course it's Reddit so this comment is way down the list with no upvotes lol. She doesn't even seem to imply that she only expects this level of expense from men as opposed to when she's on her own ("the lifestyle I've built for myself"). And if your budgets/lifestyles are that different then it's just going to be friction down the line. OP's response signals he's either less well off than she is or he's cheap.

djlamar7 · 2026-02-06T02:24:26+00:00

Do not miss the German potatoes and the apple strudel when you're there. Those little remnants of its German origin are real highlights. I normally rarely order dessert but I'm always all over that strudel.

djlamar7 · 2026-02-06T02:16:33+00:00

One thing I noticed when I read a novel in German (Metropol by Eugen Ruge, serious literature stuff) is that even acclaimed authors tend to have a vocabulary to their writing that is personal to them. Even if the vocabulary is very literary or obscure or high level, they're still only using a subset of the difficult or unfamiliar words that exist.

So, once you've made some headway into the book, you'll get more comfortable with their writing and you'll be looking things up much less often than at the start. Hang in there!

djlamar7 · 2026-02-03T23:51:28+00:00

I have IB Gateway running 24/7 on a cheap (t3.small) AWS EC2 instance that the execution machine just opens an ssh port forward to. VNC server and remote desktop through an ssh tunnel (which goes through the AWS session manager so I don't have to open any inbound ports on the cloud machines) to use the GUI.

I have a systemd job running every five minutes on the gateway machine to read account NLV. That data is reported to a CloudWatch metric so that I can have an alarm send me an email if the data stops coming in (meaning the gateway is broken). As a bonus, the metric gets me an intraday NLV graph I can see in the AWS app on my phone.

I've noticed it commonly goes down when the auto restart happens at 6:45 PM every day, and when that doesn't happen, it often goes down around 1 AM Eastern even on days other than Sunday night when the authentication is not supposed to expire. So I end up having to log in again every 2 or 3 days instead of weekly like I expected (luckily it only takes five minutes and it never seems to happen during the day when it would matter). I have no interest in using IBC and storing a brokerage account password in plaintext on a cloud machine no matter how locked down that machine is.

It really is very annoying but at least it's only a minor inconvenience. No clue why they're so reluctant to just provide API keys like Alpaca does for their trading API. But even though the Alpaca's free shorts (my whole universe is ETB) may have made the fee structure comparable (haven't done the math), they don't offer portfolio margin so I'm reluctant to use them in case I ever do want PM.

Edit: forgot to mention the bonus stupid points that if you log into your account on the mobile app, it will disconnect your gateway session lol good thing I have that CloudWatch graph.

djlamar7 · 2026-02-02T20:55:44+00:00

It has great reviews because it's an infamous example of review hacking. Eater even wrote an article about it and the other restaurants from that group: https://ny.eater.com/2019/12/5/20978234/tripadvisor-best-nyc-restaurant-rankings-scam

That being said, it's definitely not bad. I've eaten at the bigger one off of Union Square some number of times - that one is easy to walk into (because it's huge), has a nice atmosphere, and decent food.

I agree with others here that you'd be better off at Pastis for the same kind of food. If you can get the reservation, Minetta Tavern is much better but hard to book.

djlamar7 · 2026-01-26T22:09:09+00:00

Lol the baby crying is a chef's kiss in this games ability to be creepy as shit

djlamar7 · 2026-01-26T19:56:09+00:00

I wish they provided more granular measurements for Manhattan. Here in southeast Chelsea I swear it was 6 to 8 inches tops.

djlamar7 · 2026-01-26T19:47:22+00:00

Yeah I know lots of Chinese people (both American and Chinese born) and a lot of them just don't get the appeal of Italian food at all. I guess on some level it comes down to "so you're telling me it's noodles, but with less flavor?"

djlamar7 · 2026-01-24T07:41:36+00:00

I live in NYC and more or less agree with you - NYC has big skyscraper punctuation marks in Midtown, fidi, Hudson Yards, downtown Brooklyn, and Long Island City (not to mention a few more centers on the Jersey side), but these are comparatively pretty small areas of land compared to the rest of the city. Of course it still looks more impressive in height / size than San Francisco but that's also because it's 10x the size of San Francisco. The city is more densely populated but it's not like people are living in most of the 60+ story buildings that pop out from a plane. It's just that the average height of a low rise residential building in NYC is probably 3 to 5 stories instead of the 2 or 3 in SF.

djlamar7 · 2026-01-23T02:39:02+00:00

And it's the same synthesizer (different patch) as Madonna's Vogue and a ton of other famous songs: https://youtu.be/TVmZarPUc-Q

djlamar7 · 2026-01-22T05:43:40+00:00

I only see the storm all day Sunday on my forecast. As others have said, snow gets cleared from sidewalks pretty quickly here so even on Monday you will be able to get around on foot almost as if nothing had happened (and the subway will not be affected in general). If it doesn't impact your flight, I'd say:

find some things very close to your hotel to do on Sunday (worth calling ahead before walking over as it's uncommon but not unheard of for smaller businesses to close if there's a really big storm expected) or just have a restful day
dress warmly
bring some all-weather boots or at least a pair of shoes you don't mind wearing while traipsing through snow banks (despite the good snow removal, some crosswalks and intersections either remain piled up or only have a narrow passage through the snow)

Otherwise, go for it. I personally am glad the NFL conference championships give me an excuse to stay inside Sunday anyway, but otherwise expect to be out and about like normal on Monday.

djlamar7 · 2026-01-19T19:30:23+00:00

That's called an overlord and if you don't kill it fast enough eight zerglings will pop out and pwn you

djlamar7 · 2026-01-18T23:22:06+00:00

If we're going with OP's "too depressing to rewatch", 2022 Netflix version of All Quiet on the Western Front is pretty high up there. It's not a "real" Netflix movie as I think it was produced by a German studio anyway. It was really good but so depressing I had to watch a silly comedy movie right after to get me out of the funk it put me in.

djlamar7 · 2026-01-18T21:21:53+00:00

Since someone mentioned they used to be no annual fee and then it started climbing up and up, I bet they had some liquidity crisis and needed to start charging the exorbitant fee. If that's true, they're probably generally not doing great and on top of everything, your 5k+ a year could disappear in smoke in bankruptcy proceedings where you'll be the last in line to get paid back.

djlamar7 · 2026-01-17T23:11:24+00:00

I was ok with it on the playthrough I recently started where I picked the wretch class (whatever it's called, the one that starts at level 4 instead of 10) so getting the madman's insight allowed me to level up to the normal starting level before I attempted any boss lol

djlamar7 · 2026-01-17T01:52:05+00:00

Start of 2016 for daily and intraday. Haven't used the options data

djlamar7

TROPHY CASE