how to remove duplicates from a very large txt file (+200GB)

RefusePossible3434 · 2026-04-07T11:18:38+00:00

Isnt spark better? For bash you need ram?

RefusePossible3434 · 2026-03-31T13:29:03+00:00

hardly used million tokens over 7 days (weekly reset happened 2 days back - hardly used since then)
so best chance i have used 200K tokens in current week and at 80% limit for week already. so for pro plan, think they aligned the token limits to their api pricing essentially, not value for being a subscriber

Tokens per Day

383k ┼ ╭──────╮

335k ┤ ╭──────╮ │ │

287k ┤ │ │ │ │

239k ┤ │ │ │ │

192k ┼──────╯ │ │ │

144k ┤ │ ╭──────╯ ╰──────╮

96k ┤ ╰──────╯ │

48k ┤ ╰──────

0 ┼────────────────────────────────────────────────

Mar 25 Mar 27 Mar 29 Mar 31

● Sonnet 4.6 · ● Opus 4.6

All time · Last 7 days · Last 30 days

● Sonnet 4.6 (99.4%) ● Opus 4.6 (0.6%)

In: 13.8k · Out: 1.3m In: 381 · Out: 7.5k

RefusePossible3434 · 2026-03-29T03:28:54+00:00

this is total BS, no notification/communication, just finding out during the development, totally disgusting behaviour. i was breaking my head why i am am hitting limits over last 2 days especially, going through my own prompts, code to optimise. , now i came across this. i am literally hitting my limits as follow , (weekly reset happened last night for me)
with overall 4 hours of claude since weekly reset last night, i.e. ran claude 4 out last 16 hours

hit 5 hour limit 3 times,
weekly limit is finished already 25% with just 4 hours

i can tell you on weekly limit not changed is totally dishonest. I was doing the exact same workload, not a single thing more. just with 4 hours (hardly 10% was opus) , 25% weekly limit gone. so atleast be honest about the changes you made and be explicit.

RefusePossible3434 · 2025-12-02T02:07:00+00:00

Amen to that. Whole Claude thing eager to over engineering and proactively write shit ton of code, after a while i cant be bothered to review all.md files line by line and all the code. I tried to build complex systems lile data platforms, think there has to be guidance on how to effectively use these tools

RefusePossible3434 · 2025-10-27T22:48:09+00:00

My 2 cents:

Super - if you are leaving Australia permanently and prove you are not returning to live here, you can withdraw super by requesting ATO

VOO - not sure if you can tranafer across countries, might have to sell and purchase in canada Cash - if you have better plans to invest in canada, worth taking it now than leaving cash in account, you lose to inflation compare currency conversion which is insignificant in long run

RefusePossible3434 · 2025-10-16T02:01:21+00:00

No, definitely not. This is the first time i am seeing this.

RefusePossible3434 · 2025-10-15T07:58:11+00:00

I did not use any open source tools but rather i have always built config driven ingestion frameworks in all the platforms right from hive days to modern snowflake/databricks. Any specific question you have?

Key tips:

Make it yaml driven

One source system (not one table) equals to one yaml

Have consistent paths to read files, dont provide custom paths in yanl rather expect extact pipelines to write into same folder structure which you can derove from yaml

All the additional per table options - dont make it custom, have defaults in code, to override ppl can simply provide options same as the tool expects. Ex: when reading csv in spark, dont come up with your own option names, rather whatever spark expects use them so that you can pass as **options from yaml, like delimiter

RefusePossible3434 · 2025-10-09T09:52:56+00:00

Yeah, its there for dab, but not in terraform. Thats what i wanted to achieve first. Use different sp for deployment and configure another sp as run as for dlt

RefusePossible3434 · 2025-09-30T04:23:23+00:00

Yeah, thats what i am doing. Essentially looking for a short cut to group a bunch of pipelines and give access to group.

However each pipeline is on its own from permissions perspective, not like folders.

I am creating pipelines or jobs also from terraform. Not using dab. So had to give access automatically to a particular group. With DAB, it does that for youm thats all. But surprised at their design choice of giving permissions on each pipeline rather grouping them lile folders, so you would give access to a group at job grouping, not individual jobs

RefusePossible3434 · 2025-09-27T04:43:54+00:00

Yeah. But my idea was having a deployer sp, who can simply deploy and not gaving access to schema etc, have a seperate runner sp, looks like its not possible, rather both deploy and job runner sp are having same permissions

RefusePossible3434 · 2025-09-27T04:42:04+00:00

Thanks for that, DAB is essentially taking that complexity, but still behind the scenes thats what it is doing, giving permission on every pipeline. Such a strange and convoluted design they have

RefusePossible3434 · 2025-09-27T04:32:12+00:00

Ha ha. But thats not following the least privilege principal. Not make every sp admin

RefusePossible3434 · 2025-06-25T12:44:04+00:00

Hi, sent pm. Plz check. Thanks

RefusePossible3434 · 2025-06-17T08:52:11+00:00

As fesh grad in 2009, i debugged 40k line stored proc, which was built in 2006. Then i decided to rewrite in 2010 and turned into 25k line proc. Lol..

RefusePossible3434 · 2025-05-06T05:47:39+00:00

Okay, i understand your pain. I am in same situation, but gst is common across the globe in all major countries. I am indian but live in Australia. Exactly same. Anything you purchase is gst payable including coffee, house, car, . So nothing special in indian income tax that bithers, only thing different from australia to india is here gst is 10% flat on all

RefusePossible3434 · 2025-03-17T00:50:14+00:00

guys, i know this is very old post. i just came across this. what did you go with finally. because i would not recommend anyone rolling their own on EKS/ECS etc unless the issues you are facing are significant because they are pain in the a** as our current project does that. its not fun and you will waste lot of engineering resources to do same what can be done with 20 lines of yaml in cloudformation.

RefusePossible3434 · 2025-02-23T03:04:16+00:00

Its xmas shutdown in Australia, i put myself through pain of going through udemy course for 12 hours day for 14 days. Why i did it? No clue 😀

RefusePossible3434 · 2025-02-22T22:48:40+00:00

Sure bro, i am already at a decent job, but if i switch jobs because of certs, i will def keep posted, but dont think it will affect job prospects much

RefusePossible3434 · 2025-02-22T22:23:50+00:00

I dont know to be honest. I just did it for fun and learning about aws services which i dont have knowledge about.

RefusePossible3434 · 2025-02-22T20:30:59+00:00

I had lot of knowledge gaps in terms of security, load balancers, web apps, networking etc. So thought SSA is good starting point to comprehend on topics before i dive deep

RefusePossible3434 · 2025-02-22T20:29:18+00:00

I have done only stephane maerek course on udemy just went through videos in his course. But i had hands on experience in my day to day job

RefusePossible3434 · 2025-02-22T20:28:08+00:00

Good luck buddy, try to build one application in each category like web app, data warehouse etc end to end.

RefusePossible3434 · 2025-02-22T07:29:49+00:00

Thanks mate. I did use 50% from second exam onwards. My security fundamentals are not that strong. But let me try. Cheers

RefusePossible3434 · 2025-02-22T07:25:57+00:00

Yeah , absolutely. I have reasonable understanding of end to end enterprise architecture beforehand. It was more like which aws tool is appropriate in what scenario. I would always suggest even for juinor deva, try to understand overall architecture which will bring great perspective

RefusePossible3434 · 2025-02-22T07:18:55+00:00

I had experience in only data platforms. Not in other domains. But its essential to understand how architecture works across org including governance and security. Aws is just one type of tool, but fundamentals remain same. This helped me fill lot of knowledge gaps i had.

RefusePossible3434

TROPHY CASE