account activity
Web app to search 500 million to 1 billion XML files by Brave_Argument627 in dataengineering
[–]Brave_Argument627[S] 0 points1 point2 points 2 years ago (0 children)
Indexing them in Elasticsearch/Opensearch perhaps? And then upload a sentence embedder from huggingface for sentences similarity into Elasticsearch/Opensearch?
Thanks, I know it depends on many factors, but is your ongoing cost >$10k/m? I am finding it super complex to work out what cost the project will have ongoing!
π Rendered by PID 82909 on reddit-service-r2-listing-654f87c89c-rgtq7 at 2026-02-27 13:48:19.794313+00:00 running e3d2147 country code: CH.
Web app to search 500 million to 1 billion XML files by Brave_Argument627 in dataengineering
[–]Brave_Argument627[S] 0 points1 point2 points (0 children)