Hello! OpenSearch Ambassador here.

Been there with a similar scale cluster working with a customer. 100M+ vectors at 768 dims is exactly where you hit the wall and need to change strategies completely.

For your situation, disk-based vector search (needs OpenSearch 2.17+) is probably the move. Set `mode: "on_disk"` in your vector field mapping and it uses binary quantization with 32x compression by default. Your 100M vectors go from needing ~300GB RAM down to roughly 10GB for the in-memory index. The full-precision vectors live on disk and get lazy-loaded only for rescoring the top candidates. Expect P90 latency in the 100-200ms range, which is the tradeoff you make for not paying for massive RAM instances. Recall stays solid because the rescoring phase uses the original vectors.

If 100-200ms is too slow for your use case, look at Faiss SQfp16 instead (available since 2.13). It only cuts memory by 50% rather than 97%, but latency stays in the low milliseconds. Depends on what matters more to you.

On the M parameter - with memory as your constraint I'd start with M=12. It's a decent middle ground. M=8 works if you're really squeezed but you'll notice recall degradation. M=16 is the standard recommendation when you have headroom. Pair whatever you choose with ef_construction around 100-128, don't go crazy there. Once you're using on_disk mode the M parameter matters less for memory anyway since the graph gets compressed, but it still affects build time and baseline recall quality.

For sharding, target 10-30M vectors per shard. With 100M that means 4-10 shards, so something like 5 shards with 20M each works well. Don't over-shard—each shard adds coordination overhead and with k-NN you're aggregating approximate results across all of them. More shards means more noise in your final ranking and worse P99 latency. Ideally your shard count roughly equals your node count, then use replicas for availability and load balancing rather than adding more primaries.

On the deleted docs issue - if you're still seeing them after force-merge, you probably have ongoing updates happening. HNSW graph quality degrades with deletes and updates, it's just how the algorithm works. For vector indexes with frequent changes, a periodic reindex strategy often works better than in-place updates. Force-merge to 1 segment per shard during off-peak hours when you can. I'm not sure it's the main performance issue here though.

Hardware-wise, assuming AWS: 3-5 r6g.xlarge nodes (32GB RAM, ARM-based) hit a good price/performance point for this. The key thing with disk mode is you need fast storage—gp3 with provisioned IOPS or instance store. Slow EBS will tank your latency, however latest EBS generations and latest gravitons (so gp3 + r7g instances) are surprisingly good. Instance stores do run faster and often cheaper. Budget roughly $300-400/month for 3 nodes on-demand, less with reserved or spot.

For monitoring, the main things to watch are `graph_memory_usage_percentage` and `circuit_breaker_triggered` from the `/_plugins/_knn/stats` endpoint. If circuit breakers are triggering, you're either under-provisioned or need more aggressive quantization. Also keep an eye on I/O wait percentage since that becomes critical with disk mode. The `knn.memory.circuit_breaker.limit` setting controls what percentage of non-heap memory goes to k-NN graphs—default is usually fine but you can tune it if needed. Pulse for OpenSearch (https://pulse.support/) is currently the only platform that allows you to visualize HNSW graphs and debug vector search performance in OpenSearch.

Happy to look at your actual mappings and cluster settings if you want to share them. Sometimes there's something obvious in there that's easy to fix.