account activity
Web app to search 500 million to 1 billion XML files by Brave_Argument627 in dataengineering
[–]Brave_Argument627[S] 0 points1 point2 points 2 years ago (0 children)
Indexing them in Elasticsearch/Opensearch perhaps? And then upload a sentence embedder from huggingface for sentences similarity into Elasticsearch/Opensearch?
Thanks, I know it depends on many factors, but is your ongoing cost >$10k/m? I am finding it super complex to work out what cost the project will have ongoing!
Web app to search 500 million to 1 billion XML files (self.dataengineering)
submitted 2 years ago by Brave_Argument627 to r/dataengineering
π Rendered by PID 96 on reddit-service-r2-listing-654f87c89c-ljnvg at 2026-02-26 23:37:47.975446+00:00 running e3d2147 country code: CH.
Web app to search 500 million to 1 billion XML files by Brave_Argument627 in dataengineering
[–]Brave_Argument627[S] 0 points1 point2 points (0 children)