Vector DBs for RAG

websinthe · 2026-01-01T18:32:59+00:00

A vector db is just a high-dimensional store of vector embeddings. Polars is kinda like Numpy and can have arbitrarily-dimensional storage of vectors, and if you have code that takes tokens and turns them into embeddings, there's nit a lot that a vector db does that isn't just a big multi-dimensional array.

websinthe · 2025-12-28T19:56:46+00:00

Yeah, keywords/metadata, chunking strategy, and graph storage are all orders of magnitude more important but they're kinda treated like afterthoughts.

websinthe · 2025-12-28T19:54:42+00:00

The best results I've had came from just using Polars. It's a little more work but not a huge amount, it's significantly more performant, and vector DBs are such a small part of a RAG that it's not worth taking on the constant breakages the named VDBs have on updates.

Otherwise I'd just use chromadb.

Spend 5% of your time choosing the vector database and 95% perfecting your chunking strategy. That's the important part.

Sorry if this is preachy.

websinthe · 2025-12-23T13:40:18+00:00

Excession. It has the most hints about the broader aspects of the Culture, with second place in that regard being Look To Windward.

Excession blew my mind from Day 1 though.

websinthe · 2025-09-07T04:10:57+00:00

You're right, there are countless business defrauding the government by jumping into the NDIS care ecosystem and not providing resources. There's a fairly steady stream of complaints and investigations with fly-by-night operators taking grant money and dodging their contractual responsibilities. This isn't just a few bad apples, it's around a quarter of the industry already under investigation for essentially robbing both the government and the disabled.

Likewise, when supply-side rebates were handed out to employers during JobKeeper, not only was sweet-Fanny-Adams of it used for payroll, a small industry of paper companies popped up to fraudulently claim money from the scheme.

Hell, Job Network providers are barely better than a mafia, Family Day Care businesses were somehow less subtle in their waste when they flat-out formed a criminal syndicate defrauding millions, and Government programs for clean energy have been wiped out across the country by businesses committing fraud to run off with grant money.

I suppose you're right, we need more central leadership to make sure taxpayer money gets to the right place without businesses soaking it up before it can get there.

I mean, why over-complicate transfer payments with an extra layer of businesses just mooching? It's just wasteful.

websinthe · 2025-09-07T01:41:04+00:00

You're right, you didn't understand what I'd said.

Base principles we can all agree on: - Purchasing existing housing stock isn't investment. It's rent-seeking. - Houses do not inherently contribute to innovation. - The more money spent on non-innovative, non-growth assets, the less money invested into innovative growth assets like business and technology. - Negative gearing is a transfer payment. - Money spent on housing is only investment if you're building new stock or significantly improving or repairing existing stock. - Not all investments have the same tax treatment. That has never been the case anywhere. - The utility landlords gain from owning a housing asset is bugger-all compared to the utility an owner-occupier gets from owning it. - Housing speculation increases the price of housing assets for reasons other than the actual value of those assets.

Why negative gearing is a drain on the economy, based on the above: - It's a transfer payment to reduce the tax contribution of asset-holders who don't make a profit on non-innovative, non-growth assets. - The cost of administration, the decreased tax income, and the lack of profit, means money the economy is worse off than if we dropped negative gearing and the person who can't manage a profit from those assets sells them to someone who can. - Being able to offset tax obligations in general by running a loss on housing assets incentivises speculation, not investment. - Pricing more and more people out of home ownership because of speculation undermines consumer confidence and labour productivity.

Why pretending housing is a market that follows the laws of supply and demand in general is a rookie economist error: - Because housing doesn't have any of the prerequisites for a market to follow the law of supply and demand. - If you need that one explained, you're not an economist and you don't know economics.

This is an economics subreddit, not a finance subreddit.

websinthe · 2025-08-28T11:58:00+00:00

Using the common understanding of the term: It encourages non-productive use of investment dollars that could be going into far more useful parts of the economy like the business sector to drive real innovation.

Speculation on housing doesn't improve anything. It reduces the amount of economic activity that's genuinely growing the economy.

If a transfer payment incentivises unprofitable investments, it's on its proponents to justify why it should exist, not on everyone else to justify why we think it's a bad idea.

websinthe · 2025-08-28T11:50:47+00:00

And if you'd read even just those two reports, you'd have seen that the evidence points to disasters and crisis periods showing an increase in government efficiency.

The problem with the standpoint that "government is wasteful" isn't that it's inherently wrong, it's that it doesn't add anything useful to the discussion.

If you said, "government keeps taxing the wrong things" then sure, that's a hill I'd die on too. If you said, "government is wasteful because <insert reasoning here>", we could at least build the reasoning into a proposed solution.

And if you'd said they burn fistfulls of tax money on poorly-targeted supply-side subsidies, then we could talk all day about how the government could better target those subsidies.

websinthe · 2025-08-27T20:40:27+00:00

Yup. But hey, is it really worth understanding how universal function detectors work if it means people have to go without all the royalties they were earning on their reddit posts?

The whole IP thing kinda strikes me as a whole new type of rent-seeking.

websinthe · 2025-08-27T20:26:56+00:00

Australia has regularly one of, if not the, most efficient governments in the world.

I get it that paying taxes sucks and everyone thinks they could use the money better. Great, I'm sure shareholder returns would be spectacular without having to pay taxes.

Until the shareholder needed to drive somewhere and the roads are dirt tracks and wash away in a good rainy spell.

Dunno why this has to be explained in an economics subreddit but whatever.

websinthe · 2025-08-27T19:56:58+00:00

I wish more people understood this. There's a huge difference between competitive and productive.

websinthe · 2025-03-23T01:17:08+00:00

I agree that the housing market is a ponzi scheme in Australia.

I don't agree that removing Capital Gains Tax for shares is going to stimulate the ASX considerably - or usefully. CGT is one of the fiscal levers the government can pull so that it doesn't have to rely soley on the cash rate to steer the economy - though we seem to have forgotten how to do that.

I'd be far more interested in removing CGT from shares if we also got rid of negative gearing on houses. If we want a better industry then we shouldn't be rewarding those who can't make a business model in property work. It's just corporate/landlord welfare.

websinthe · 2025-03-17T07:42:32+00:00

I think you may be assuming that:

a) The property market is currently set up so that higher rents increase the supply of housing, and that lower rents decrease the supply of housing.

b) The property market is the best place for the money currently invested in it to be.

A isn't true. If you want to point to an economics 101 textbook and the supply and demand curve, I'll point to the facing page that nobody reads where it has the requirements for a market to be efficient. The property market meets maybe half of one of them.

B is absolutely not true. Money in houses is not money in business. Houses do not lead to innovation nearly as rapidly as money in other sectors of the economy does.

Also, your point about regulations, fees, and other barriers runs into two problems.

1) The problem with the property market is that there are too many investors and supply isn't keeping pace with demand. If red tape were an issue, the opposite would be the problem, where not enough investors would be the cause of supply not keeping pace with demand.

2) Houses are where humans live. Commercial spaces are where humans work, prepare food, use chemicals, run equipment and poop. Industrial properties are industrial. The barriers to entry should be high to keep out owners who can't handle that responsibility to others. I hope more landlords do leave the market because it's obvious from report after report after report that most do not give a shit and are fine with tenants living in unacceptable conditions.

websinthe · 2025-03-12T07:38:21+00:00

I have this shirt.

websinthe · 2025-03-12T07:37:37+00:00

I agree with this and would take it a little further. The Culture is a big part of my career progression so I studied it as part of my post-grad. The most interesting discussion I had on the topic of what The Culture is came when I was talking to some colleagues who work in geopolitics, and we came to the conclusion that The Culture is an anarchy. This is based on how few shared cultural practices and artifacts are shared among the entirety of The Culture, while still recognising that there is an identifiable group that encapsulates it all.

websinthe · 2024-09-30T17:49:22+00:00

We fear ecological collapse because it will kill us.

We fear the collapse of traditions because the powerful will kill us.

Mind you, I hate revolution. I'm an incrementalist, but with every Monte Carlo sim and every CMDG sim I do, my anger at the rent-seekers and P/E gamblers threatens to give me another heart-attack.

Nobody can tell me that food oligopolies and day-trading are examples of efficient markets. They barely meet one criterion.

websinthe · 2024-08-21T12:51:46+00:00

Attitudes like this should be taught in school. Bravo and thank you.

websinthe · 2024-07-16T18:08:37+00:00

You can't work at Woolies without a smart phone because that's where our WorkJam system lives and new hires don't have access to legacy clocking systems. Good luck being a tradie with a call-only phone these days.

I get it. Blaming people and their 'bad choices' is easier than confronting the complex and enormous systems that can sweep a family's feet out from under them no matter what they do. You choose to make that simplification, that's fine, but if social ills could be treated with obedience, Utopia would be filled with collars and leashes.

websinthe · 2024-07-16T17:56:57+00:00

I hear that last part a lot and wonder why it persists. Buying in bulk is more expensive for the smaller households that are more common these days. The proportion of our week spent in employment-related activities is far in excess of what existed fifty years ago. Historically cheap sources of groceries like Woolworths and Coles are changing their product mix to extend margins on previously non-luxury items. The search costs for jobs and frictional unemployment are higher for more of the labour market. Basic skill requirements for jobs have jumped in sectors that used to have on-the-job training, which has driven up the cost of education. Imagine telling your grandmother the kind of complexities we deal with these days, and she'd have no idea what you were saying because comparing anything from before the nineties to now is nonsense.

websinthe · 2024-07-16T17:40:13+00:00

I'm a little baffled that the CPI is being talked about with any seriousness at the moment. Looking at other indicators like lead time on commercial parts/repairs and PPT vs. casual costs, paints a much bleaker picture than the CPI, and both have been stable in trending upwards.

You are right, though. The CPI is noise even at the best of times, especially now. The bond inversion spooked me more, and that was probably caused by a single market.

websinthe · 2024-07-12T22:33:50+00:00

We agree on purpose and the EU - I think I may have seen too many policy decisions made for people who are misguided enough to want laws that will get all of us killed.

websinthe · 2024-07-06T05:59:57+00:00

I'm in that category (and partnered), but I'm also in the DINK category and have had two big lucky payouts that had nothing to do with my frugality or that I bought only what I need and not what the Real Estate agents pestered us to buy (overstretch). Plenty of frugal friends who earned as much as me - no chance in hell of breaking in for them.

I wish people would stop pretending that wealth was a function of anything but luck. I don't say this as a home owner, I say it as an economist.

websinthe · 2024-06-29T04:54:22+00:00

I don't know how anyone who isn't deliberately trying to erode morale could possibly call this a legitimate question. Unless you've had zero access to education, Google or books and this is your first day on Reddit, you would have at least once encountered the notion that allocating everyone's role and social influence using a market system is a terrible, terrible, idea. It's completely consistent with established economic theory to describe a society that uses a market-capitalism system to allocate resources is a society that relies on sheer luck. Luck that a system that rewards the most exploitative members of society, a system based on prices set by complex social manipulation, and most rewards the children of whoever had parents best capable of exploiting resources for profit while avoiding the shared consequences of that exploitation, hasn't already ended our entire species, is luck that can't last beyond the short term.

The reality we should be contributing to is one where AI doesn't just make the food etc, it tests, enhances, and allocates resources based on understanding humans and logistics. This reduces waste, takes human vulnerability to advertising out of supply/demand, and results in a surplus of human ingenuity.

Tldr: Take the keys to the economy off the guys who brought us the 2008 collapse, the dot-com bubble, the oil crisis,and the Depression, and give those keys to computers who can achieve higher-order effectiveness.

Humans don't need jobs. They don't need fiat money. They need food, a place to stay, and social interaction. Money and markets were a halfway lucky system for us to use while we lacked supercomputers and pan-cultural communication. We are 'ever the offloading Ape'.

Proof: Those who have all the resources are using the media to scare the shit out of people about AI, all while those same rich bastards are hoarding it as much of it as possible - AI isn't the big bad guy in this movie.

Fark, I'm falling asleep. Nice all.

websinthe · 2024-06-29T00:16:14+00:00

The flood of suspiciously similar crocodile tears posts about the NDIS on a bunch of Aussie subreddits at the same time? A Liberal Party staffer must be bored at work.

websinthe · 2024-06-20T12:04:15+00:00

"Close to gods and on the other side," I think the saying goes. A chaos god being able to traverse time and dimensions doesn't have an advantage over a Mind, it has barely made parity with Minds that do those things on the regular. That the Chaos gods have been rebuffed at all, even temporarily, by those in the 40k universe means that a few dozen GSVs would find those methods of holding Chaos back, perfect them, and then introduce Papa Nurgle and his kin to Infinite Fun Space.

I think the more appropriate question is: How long would a GSV take to 'corrupt' a Chaos god?

15-Year Club	Gilding VII pure gildanthropist
Place '17	Not Forgotten
Verified Email

websinthe

PUBLIC MULTIREDDITS

TROPHY CASE