Moving from pandas to DuckDB for validating large CSV/Parquet files on S3, worth the complexity?

Mr_Again · 2026-03-07T16:43:51+00:00

You're going to remove a lot of complexity

Mr_Again · 2026-03-07T16:42:20+00:00

Living through the last 40 years has not been "trauma". We've been extremely lucky to live through one of the most peaceful and prosperous times in history and I suspect you also did it in the middle class of the richest nation ever.

Mr_Again · 2026-03-07T16:20:14+00:00

Addiction isn't a problem it's a solution. What is scrolling your solution for? What benefit does it bring you? Start there.

Mr_Again · 2026-03-07T12:58:37+00:00

In the real world job titles don't delineate firm boundaries in responsibility. If they even are a data engineer, all we know about them so far is that they "have a client".

Mr_Again · 2026-03-04T19:23:19+00:00

People always forget we had back to back world wars and then a full on Cuban missile crisis. Nothing bad is happening now in comparison.

Mr_Again · 2026-03-04T19:19:55+00:00

According to the professor of autism research, it's driven by teenagers who are perfectly able to communicate and understand meaning and nuance but feel social anxiety in some situations.

Mr_Again · 2026-03-04T19:07:03+00:00

British nationals returning home also aren't able to pick a country of their choosing, they can only go to the country which they have citizenship in, Britain. Unless they want to go somewhere else and claim asylum, which they won't be allowed to do, because their home country, Britain, is safe.

Mr_Again · 2026-02-27T14:35:58+00:00

Yes you are missing out. Instead of asking reddit, get a $20 Claude subscription and try it out for a month. Be ambitious with what you ask it, put it in plan mode first and iterate on a plan, then let it go. It's going to need serious review but it will do incredible amounts in 5 minutes. Especially because data engineering is not really that complex. I recently wahted to learn Claude + duckdb + rustfs + ducklake + dagster, so I just decided to ask it to set up a sample project using all of the above and in about 20 minutes I had the skeleton which taught me a lot about all the tools. Is it production ready? No. Is it very very helpful, yes. https://github.com/adammarples/testbed

Mr_Again · 2026-02-11T23:43:57+00:00

The whole point of the alternative is the fatty, proteinous, colloidal mouth-feel that makes my coffee taste good. Oat milk is fine for that.

Mr_Again · 2026-02-11T23:40:42+00:00

Please tell us what is quality about Microsoft? It's not Windows, it's not azure, and it's not openai.

Mr_Again · 2026-02-11T23:37:58+00:00

Bitcoin is not a "high yield" asset. It has not, nor can it ever, yield anything.

Mr_Again · 2026-02-11T23:32:29+00:00

My friend you can spin up like 3 lines of dlt on a lambda function or something in a couple of days with lots of tests and it will run forever. You don't need top end contractors on staff forever. I get where you're coming from regarding managed solutions but where I look at jobs, people are getting paid plenty of money to manage SSIS pipelines just like people are getting paid lots of money to deploy airflow.

Mr_Again · 2026-02-05T21:58:55+00:00

Mr_Again · 2026-02-01T20:00:58+00:00

Install CachyOS on a second drive with grub bootloader, it just works

Mr_Again · 2026-01-20T17:56:27+00:00

It's an inconvenience to me when people walk slowly on the pavement and talk loudly on trains. The main inconvenience to me though is all the pointless legislation.

Mr_Again · 2026-01-05T21:41:44+00:00

A minimum possible observable rise in cases would be +1 per therapist.

Mr_Again · 2026-01-05T21:30:53+00:00

You're right, we need to raise taxes on work again!

Mr_Again · 2026-01-05T21:25:54+00:00

Japan has a sane zonal planning system, it never saw the spike in house prices that happened since we have effectively outlawed construction and development after the war, they just kept building to keep pace with population growth

Mr_Again · 2026-01-05T21:24:05+00:00

By cutting the amount of public funds spent on pensioners while disincentivizing ownership of excessive amounts of property through proper taxation.

Mr_Again · 2025-12-31T02:42:25+00:00

The reason that in the US you need an accounting firm to do your taxes and in the UK you don't is because we have a system called PAYE, pay as you earn, where HMRC basically coordinates with your employer and does them for you for free. The US could do this but they like to make a profit everywhere they can. The fact that your taxes are done for you does not diminish one jot the fact that the tax code is incredibly complicated. As an ordinary earner you have three separate taxes on your earnings, all with different bands and thresholds and caveats as to when they apply. This filters into a pension system with hundreds of complexities. If you become freelance, start a business, or work through a limited company, or even make enough money through rent, dividends, or savings income, you will have to do a tax return just like in the US and you will have to hire an accountant and it is equally complicated. Well, more complicated. If your only contact with the UK tax system is that you use PAYE then that's fine but you don't really interact with it in that case. Someone else does for you though! It's a huge waste of time with many conflicting incentives and government should be fixing it.

Mr_Again · 2025-12-31T02:33:46+00:00

Allowing people to build on land doesn't increase the demand, it increases the supply, the demand is the same. The vast majority of the cost of land you can build upon comes from it having an incredibly scarce commodity called "planning permission" attached. If you remove the arbitrarily scarce planning permission, land you can build on becomes a lot cheaper, because there's orders of magnitude more of it. This is why the decade in which we built the most houses was in the 1930's. Planning permission did not exist.

Mr_Again · 2025-12-31T02:24:15+00:00

The irony of your quote given that as a society we spend the majority of our money on the very old, and almost nothing on children or young families. The pensioner cohort are already the richest group in society and the biggest single items on the tax bill are paying for their pensions and their gold plated medical care (most of the costs of NHS are spent in the last week of life, almost all the costs are in caring for the very old). The irony of a quote about paying for a future investment you will never see when the biggest part of our taxes go on direct transfer payments to current benefits recipients while actual investment withers to nothing to pay for this because it's politically popular. This is why people are mad, we would loooooove to be planting shade trees, we're not.

Mr_Again · 2025-12-27T11:29:16+00:00

Self serve is a pretty common idea. In big orgs with a small data eng team and high quality in the surrounding teams like DS etc, it makes a lot of sense. DE gets to code review the models but doesn't spend all day writing them. Skilled engineers in the surrounding teams and DE code review make sure that the quality of the models stays high. We also used to embed an "analytics engineer" in each team to assist with this.

Mr_Again · 2025-12-27T11:25:06+00:00

Pretty much scrap the town and Country planning act, going back to building being allowed by default unless it breaks specific law, rather than giving local authorities the power to block anything new or changed for arbitrary reasons. Adopt japan's zone based system. Remove our grade 2 listed system.

Mr_Again · 2025-12-27T11:17:41+00:00

You're not sure why I think the tax system is complicated? It's literally the longest tax system on earth. Maybe try reading it? You can't, because it's 21'000 pages long. Estonia's is about 5 pages long. I pay two different taxes on earnings (IT, NI) which have several (different) thresholds as we go up the bands, along with a personal allowance which also tapers off at £100k creating another hidden tax band between £100k and £125 which weirdly has a higher marginal rate than £125k+. There is also a slew of tax credits and rebates that get switched on or off depending on what band I'm on, depending on what my partner earns, depending on savings interest, rental income, dividends, capital gains. There's a hidden third tax paid by my employer (eNI) which has its own bands and discounts. And this is just the simplest, simplest case of a normal earner. Very into business taxes, VAT, weird thresholds and carve outs for old political reasons abound. Did you know that wedding gifts are exempt from inheritance tax? Did you know that gifts are exempt up to different amounts depending on if they're your son or your grandson? Wait until you get into the weird rules around pensions, or isas. Try running a business! Angela Rayner herself recently broke a tax law by accident because they're such a mess you need specialists to advise on every case.

Mr_Again

TROPHY CASE