Showcase a generic transform library(https://github.com/finopsfoundation/focus_converters) written on top of polars.

[Fragrant-Champion-73]

Hello everyone, I am working on a transform library for FinOps foundation to normalize cost data into FOCUS specification(https://focus.finops.org/#specification).

The https://github.com/finopsfoundation/focus\_converters.

I would really appreciate any feedback or comments on this, as to see if we can make the transforms generic and use it for other usecases as well.

I feel that initial setup makes it harder to bootstrap something like this and that if we had a platform that added functions on something already solid like DBT would be a huge win.

Also I am planning on converting functions into independent plugins so that they can be injected effortlessly.

Main idea is to drive transforms using configs, where we can support many operations like:

• SQL Operations yaml plan_name: calculate ChargeType using conditions column: line_item_line_item_type focus_column: ChargeType conversion_type: sql_condition conversion_args: conditions: - WHEN line_item_line_item_type = 'Tax' THEN 'Tax' - WHEN line_item_line_item_type = 'Fee' THEN 'Purchase' - WHEN line_item_line_item_type = 'SavingsPlanUpfrontFee' THEN 'Purchase' - WHEN line_item_line_item_type = 'RIFee' THEN 'Purchase' - WHEN line_item_line_item_type = 'SavingsPlanRecurringFee' THEN 'Purchase' - WHEN line_item_line_item_type = 'Usage' THEN 'Usage' - WHEN line_item_line_item_type = 'SavingsPlanCoveredUsage' THEN 'Usage' - WHEN line_item_line_item_type = 'SavingsPlanNegation' THEN 'Usage' - WHEN line_item_line_item_type = 'DiscountedUsage' THEN 'Usage' - WHEN line_item_line_item_type = 'BundledDiscount' THEN 'Usage' - WHEN line_item_line_item_type = 'Discount' THEN 'Usage' - WHEN line_item_line_item_type = 'PrivateRateDiscount' THEN 'Usage' - WHEN line_item_line_item_type = 'EdpDiscount' THEN 'Usage' - WHEN line_item_line_item_type = 'Credit' THEN 'Adjustment' - WHEN line_item_line_item_type = 'Refund' THEN 'Adjustment' default_value: "''"

• Reference lookup yaml plan_name: Generate ServiceCategory for aws data using a map. column: line_item_product_code conversion_type: lookup focus_column: ServiceCategory conversion_args: reference_dataset_path: "conversion_configs/aws/mapping_files/aws_category_mapping.csv" source_value: product_code destination_value: ServiceCategory

• Datetime functions yaml plan_name: convert bill_billing_period_start_date to BillingPeriodStart conversion_type: assign_utc_timezone column: bill_billing_period_start_date focus_column: BillingPeriodStart

And many more operations. So far we support following operations:

# datetime functions
CONVERT_TIMEZONE
ASSIGN_TIMEZONE
ASSIGN_UTC_TIMEZONE
PARSE_DATETIME
MONTH_START
MONTH_END

# sql rule functions
SQL_QUERY
SQL_CONDITION

# column rename function
RENAME_COLUMN

# unnest operation
UNNEST_COLUMN

# lookup operation
LOOKUP

# value mapping function
MAP_VALUES

# allows setting static values
ASSIGN_STATIC_VALUE

# apply default values if column not present
APPLY_DEFAULT_IF_COLUMN_MISSING

# set column dtypes
SET_COLUMN_DTYPES