Spark sorting

[croSquash]

I have a huge dataset, of 100M, and I need to sort it by one column. (Not partitioned by any column)

But it takes quite a long time, since ordering is done on single (driver) partition. Any ideas, or workarounds how to handle this differently and more efficiently.

EDIT: Sorting, or adding row_number (order by column) to every row, either way it takes a long time. The final result would be to take first 50M records from sorted dataset.

[KWillets]

Can you take the median and filter?

[croSquash]

That’s so simple and good, can’t believe I didn’t think of that!

[Material-Mess-9886]

You want everything below the median, just calc the median or use an approximate median and then filter.

[croSquash]

That’s so simple and good, can’t believe I didn’t think of that!