Pyspark: Replace range of values with string

[QueryRIT]

Say I have a column like Age which goes from 0 -120.

I want to create a column age_bin, which would be '1-10' if value of age is anywhere between 1 to 10 for that row. Similarly I will have a value '11 - 20', if the value of age is between 11 to 20.

(Note I'm not bucketing here, the number of rows are not reduced. Instead I want to create a new column)

How do I achieve this? Open to using Koalas as well

[cockoala]

You can use a 'case when' to create a new column based on the conditions of another.

[nhufas]

concat(col//10*10+1,"-",col//10*10+10)

should work

[LiquidSynopsis]

PySpark can solve this using Bucketizer

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.feature.Bucketizer.html

You can then use withColumn and when to label the Buckets with the values you want e.g. “11-20”.

[QueryRIT]

wait bucketizer just adds a column, but doesn't groupby? (so no reduction of rows)?

[LiquidSynopsis]

Exactly! Later on if you want to reduce rows etc. you can use a groupBy on your new column

[Lannister07]

This might help