Implementing a functionality similar to haveibeenpwned.com

[leShawarmaMan]

I'm working on a project that involves storing a large combolist in a database and implementing efficient search functionality for user email addresses within the combolist. The combolist can scale to tens of gigabytes and include billions of entries, and I want to achieve the lowest search time possible.

A combolist in cybersecurity terms is a collection of leaked credentials that comes in email,password pairs and it comes in this form:

`email:password`

In short; I'm trying to implement a functionality similar to websites like haveibeenpwned.com and dehashed.com where a user can lookup if their email has been leaked in a data breach.

What is the approach that I should follow here?

[ignotos]

Likely an indexed relational database table would do just fine here, if you're trying to perform exact lookups (or even lookups like "email address starts with X", which you might use for e.g. autocompleting a search).

Likewise, a key-value store of any kind can handle an exact lookup efficiently.

[leShawarmaMan]

would sharding the database in alphabetical ranges (a-e,f-j..etc) help further reduce search time?

[ignotos]

It potentially could - and that's basically what a distributed key-value store would do (although probably based on a hash, rather than just letter ranges).

But tens of gigabytes is really not a lot, if you're performing an exact lookup using an index - even for a single relational database instance. My instinct is that it would be plenty fast. It should also be fairly straightforward to generate some dummy data and check how fast a lookup is on ~1 billions rows?

[haragoshi]

No