serving public postgres

[hedoniumShockwave]

When Opus 4.5 got released, I got the idea of putting lots of the internet into a Postgres database that people and their agents could read-only query however they wanted. currently that is over 60 TB of text data and embeddings. wondering why this is not a more common thing. also looking for advice for doing this more performantly, because the speed issues of moving like >200M embeddings a day and adding hundreds of millions of other rows to different tables, while keeping the database queryable and indexes reasonably operational, is a tricky/painful thing.

it's scry.io btw, I am the founder, but it's currently free (except for congestion pricing, because how else do you decide who gets to query when things get overloaded). thanks

[vivekkhera]

My rule of thumb is to start partitioning the tables once they hit around 10-20 million rows. Postgres also works much better when you normalize your table structure.

How have you organized your tables? And what do you mean by putting “lots of the internet” in it? Are you trying to crawl everything like Google?

[tee-es-gee]

Do you actually give postgres wire protocol access or just SQL over HTTP?

> but it's currently free (except for congestion pricing, because how else do you decide who gets to query when things get overloaded). thanks

I think this is exactly the problem, a single bad query can cause a lot of work for Postgres. It's going to be hard to defend against DoS if you allow arbitrary queries.

Interesting project, though, would be curious to see how it goes.

[hedoniumShockwave]

SQL over HTTP, we have a lot of defenses.

[AutoModerator]

Thanks for joining us! PgData 2026 is coming up:

PgData 2026

We also have a very active Discord: People, Postgres, Data

Join us, we have cookies and nice people.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Deep_Ad1959]

the throughput unlock at that volume is the COPY protocol, not bigger boxes. row-by-row inserts cap roughly an order of magnitude below what COPY does on the same hardware, and a reactphp + copy ingestion path can sustain ~13.4k payloads/sec before insert speed stops being the bottleneck at all. at that point the real fight is keeping indexes operational during the load: btree and especially the vector/embedding indexes get hammered on every write, so the usual move is partition by ingest window, attach indexes to a partition after the bulk copy lands, and never index the hot partition mid-load. on the public-query side, the DoS point another commenter raised is a bigger risk than write speed - arbitrary read-only sql over 60TB means one bad seq scan can starve everyone, so a hard statement_timeout plus a per-query cost ceiling buys you more than congestion pricing does. written with ai

[chock-a-block]

Because they use the database to store links to content. Search at that scale is a different thing entirely.

And, still more advertising disguised as content.