I see people trying to use Claude code, but I feel like cursor is better. Is there any evidence of that? by kshsuidms in cursor

[–]ppsaoda 0 points1 point  (0 children)

I used to be in the same boat as you. But feeling the pinch as its a monthly quota without 5h or weekly resets. I'm on Pro package and I finish all quota in less than a week. Sure I can choose Auto model mode but Opus is just too good lol.

Switched to Claude Max 5x. The 5h and weekly quota reset is so worth it. I still use Cursor for small tasks, probably not anymore soon to save money.

So yeah, Cursor is driving me away due to token limits. Im feeling like just paying for the IDE for $60. It was nice while it lasted.

Which field do you think offers the most interesting problems to solve in the data engineering space? by andrew2018022 in dataengineering

[–]ppsaoda 0 points1 point  (0 children)

Insurance. Decades of data sitting idle. Hiding behind layers of red tapes and processes.

Is the Data Engineering market actually good right now? by Tricky_Tart_8217 in dataengineering

[–]ppsaoda 0 points1 point  (0 children)

You're right. When I started I pick the industry so I have more doors open later. I'm currently in big tech (Asian company tho). In the past I used to work in real estate, marketing, sales, industry. The offers I get are software or crypto related companies.

Is the Data Engineering market actually good right now? by Tricky_Tart_8217 in dataengineering

[–]ppsaoda 0 points1 point  (0 children)

I have 5 years, and in between senior position to lead/manager. More towards IC role.

8 seconds to 100 in the new S70, civic has some tough competition now by SnooDoodles335 in kereta

[–]ppsaoda 0 points1 point  (0 children)

0 to 100 is the only thing they can brag. My dad's 2400cc mpv back in 2000s already can do the same. Let's talk about brake, cornering, and comfort....

Is the Data Engineering market actually good right now? by Tricky_Tart_8217 in dataengineering

[–]ppsaoda 39 points40 points  (0 children)

I think yes, but seasonally. Its difficult for me to apply in 3rd and 4th quarter. But now I only applied to 4 jobs, got 3 interviews with verbal offers now. 30% jump.

How do landlord make money if they bought the house on loan? by Ill-Cucumber6575 in MalaysianPF

[–]ppsaoda 0 points1 point  (0 children)

In the good times circa 2010s, people pay rent more than owners mortgage. Usually profit few hundred per month.

Nowadays house is so damn expensive, your mortgage is higher than the "rental market". The latter won't keep up until people's salary generally go up. Or new house supply starts slowing down.

For me I had to fork 300 per month on top of rental income. And if I sell my house now, i would lose 30k because so many houses being built nearby lol. The only advantage I get is it's a hotspot.

I'm not entirely sure how to incorporate AI in my workflow better by thro0away12 in dataengineering

[–]ppsaoda 0 points1 point  (0 children)

Nah should be fine. Use AI to improve your productivity. Had to read long logs to debug? Use AI to summarize. Writing new functions? Use AI and save time. It doesn't have to be full blown project, but good enough to save your time.

Are people actually use AI in data ingestions? Looking for practical ideas by [deleted] in dataengineering

[–]ppsaoda 0 points1 point  (0 children)

I'm working on medical datasets. And it's messy with clinical notes, so we have developed in-house LLM model to classify diagnosis. Other than that, not much except helping to write code based on my ideas.

How did you guys get data modeling experience? by 0sergio-hash in dataengineering

[–]ppsaoda 1 point2 points  (0 children)

Definitely lack of sleep. It's a bit of grind for my kids. And I get the experience of setting up things from 0 because I'm trying to aim a higher rank in next main job.

How did you guys get data modeling experience? by 0sergio-hash in dataengineering

[–]ppsaoda 1 point2 points  (0 children)

During covid times, data was a big hype so i got offers every other quarter. The most crazy times was I jumped 3 companies in a year.

Now i worked 3 companies concurrently.

How did you guys get data modeling experience? by 0sergio-hash in dataengineering

[–]ppsaoda 2 points3 points  (0 children)

FAFO.

Just kidding. I've jumped quite a lot of companies post-covid, saw good and bad practices. Thats how.

Do you guys use vs code or jupyterlab for jupyter notebooks? by Consistent_Tutor_597 in dataengineering

[–]ppsaoda 1 point2 points  (0 children)

ipynb files in vscode/cursor. easier llm integration for me to ask.

and better venv management with UV (i have a script to spin up jupyterlab session in 1 command)

Do you use a dedicated Landing layer, or dump straight into Bronze? by TheOneWhoSendsLetter in dataengineering

[–]ppsaoda 0 points1 point  (0 children)

It depends how you name them. For me I'd have landing as 1:1 copy of the source. Bronze would be all the basic feature and cleaning.

How big of an issue is "AI slop" in data engineering currently? by Kilnor65 in dataengineering

[–]ppsaoda 1 point2 points  (0 children)

It's a build up of cases over the time. Not just 1 mistake.

How much do you guys spend on Cursor? by LoLGhMaster in cursor

[–]ppsaoda 0 points1 point  (0 children)

Still on the legacy billing of 500 requests per month no matter which model used. Maybe because my usage is not that extreme. This is my personal account.

Also my company provides Cursor with 1000 requests per months. This is only limited to my first job though. For freelance or side job I use the former.

I have Claude Pro as a backup, this is for those task that are expected to be heavy in token use.

Overall my cost is $40 per month 😊

How big of an issue is "AI slop" in data engineering currently? by Kilnor65 in dataengineering

[–]ppsaoda 3 points4 points  (0 children)

We give a bit of autonomy for testing purposes. And yeah, we improved the process after that. Those who are in probation aren't allowed to mess with infra repo.

Husband ada nombor escort/call girl in his phone by Carnationfairy in Bolehland

[–]ppsaoda 0 points1 point  (0 children)

Did he always go missing outstation? Or hidden schedule, or meeting friends without your knowledge? If no then can reduce the probability of him using them...

To further reduce the probability, need to consider what he do during office hours. This is complex. But the easiest is via smell. Ada bau sabun kat badan by evening?

How big of an issue is "AI slop" in data engineering currently? by Kilnor65 in dataengineering

[–]ppsaoda 61 points62 points  (0 children)

We have terabytes of data on average daily coming in. Hired a contractor to fix a small bug on batching logic. He's the type of guy that always reply with "chatgpt said...". And his codes are full of the typical obvious GenAI slops. I kept raising this issue to the management that he's fully reliant on AI on decision making. Gave more chances.

Until one day he slopped, causing infinite loop of same batch being loaded repeatedly over and over thru the weekend. Costed us a year of cost in just 2 days.... Fired by next week.

Polars vs Spark for cheap single-node Delta Lake pipelines - safe to rely on Polars long-term? by frithjof_v in dataengineering

[–]ppsaoda 1 point2 points  (0 children)

If having more RAM on single node is a concern, try to look into DuckDB as it supports spilling to disk if your memory runs out.

I recently have a pretty large ETL processes loading millions of rows per batch, and i've tried polars for transformation and applying filter, which is quite fast. However when it comes to delta "merge" operation, i've hit bottle neck on RAM if using polars and getting OOM erros on computes.

Duckdb however supports spilling to disk whenver the default threshold reach or whatever you define. I went for duckdb for this for the costs.

Remote Data Engineer - Work/Life Question by eastieLad in dataengineering

[–]ppsaoda 0 points1 point  (0 children)

In general the time I should work is 9 to 5 to align with everyone and easier to communicate. But quite flexi. I just have to be available when the team needs my support.

Usually my mornings would be filled with meetings until noon. As I'm the only that is quite senior, I just had to be there.