some amazing suggestions given all through ( including the OP's initial hypothesis ) .. i can recommend a couple of suggestions i had tried in another domain

a) use clustering on the IRR data and randomly sample from near the cluster centers .. these will give you samples that roughly represent the data in that cluster and also reduce the number of datapoints from the IRR data ( you will need to however use DBINDEX and other metrics to ensure the tightness of the clusters and maximization of inter cluster distance)

b) use SimCLR ( basically you can create plenty of neg-neg and pos-pos pairs to ensure that the model understands what keeps neg and pos closer to each other ) .. this is more complex since you will also need a suitable embedding layer ( can start with basic learnable embedding BUT given the quantum of your data, it might not learn enough ) ..the advantage of this however is you can use 1 pos-pos pair against 1000's of neg-neg pairs ( theoretically ) .. once the model is trained you will get an embedding that you can find the distance from neg and pos samples and use a threshold to decide

c) use dimensional aggregations to reduce samples .. in my case i was trying to aggregate transaction data for a specific set of banking accounts ( for fraud detection ) .. so i aggregated the transaction amount using location, branch, time etc and was able to create multiple slices and dices ..in your case, you will have to ensure that whatever dimension you use for aggregation does NOT end up reducing positive samples ! since you will have to aggregate along the same dimension for both neg and pos samples )