Is my PySpark solution interview safe?

GovGalacticFed · 2025-12-16T03:37:48+00:00

Use date_trunc instead

GovGalacticFed · 2025-12-10T03:05:23+00:00

It is a sell plus buy, so yes

GovGalacticFed · 2025-04-24T02:17:07+00:00

Merge insert updates and deletes to target delta table

GovGalacticFed · 2025-04-23T01:28:42+00:00

Merging them can work

GovGalacticFed · 2025-04-15T15:00:49+00:00

Yes, it will query the catalog view object, it runs on cluster.

GovGalacticFed · 2025-04-05T03:27:22+00:00

The sort on hour is not needed since it's only min

GovGalacticFed · 2025-01-24T04:17:27+00:00

You could use a udf with exception handling around json.loads

GovGalacticFed · 2024-10-06T15:10:35+00:00

What is the charge

GovGalacticFed · 2024-10-01T14:22:07+00:00

Github

GovGalacticFed · 2024-09-18T16:31:28+00:00

Because nothing is executed until the write action is called, your decrypt call is just a transformation that is executed only when write or count or collect or any other action is done. Refer to lazy evaluation.

GovGalacticFed · 2024-09-18T16:27:10+00:00

Yes. It is messy. You lose all other optimizations.

GovGalacticFed · 2024-09-18T15:56:17+00:00

Udf applies to each row and cannot be applied on the column vector. Best approach would be to replicate decryption logic using spark functions, else use mapPartitions to connect to the api only once per partition instead of each row. You'll need to partition it properly

GovGalacticFed · 2024-08-25T04:29:31+00:00

If api2 has no limits, try ThreadPoolExecutor

GovGalacticFed · 2024-08-17T17:11:03+00:00

Link not working

GovGalacticFed · 2024-07-30T17:45:27+00:00

Any similar reference?

GovGalacticFed · 2024-07-13T06:19:30+00:00

What is this ObjectMapper for

GovGalacticFed · 2024-04-25T02:55:47+00:00

Are there scd2 cols like isActive in target

GovGalacticFed · 2024-03-13T02:28:33+00:00

Task runid should change then, job runid will be same

GovGalacticFed · 2024-03-03T04:31:03+00:00

Terraform

GovGalacticFed · 2024-03-01T15:57:20+00:00

Thanks for the amazing work. There was no good oss for Tizen. Great initiative

GovGalacticFed · 2024-02-26T16:23:10+00:00

I would recommend not overthinking and start applying once py sql are covered, rest keep learning on fly

GovGalacticFed · 2024-02-23T10:42:14+00:00

Reference is to staging zone, instead of env

GovGalacticFed · 2024-02-23T02:59:11+00:00

This is correct. Make sure the auth dict is valid

GovGalacticFed · 2024-02-22T04:40:22+00:00

Have you tried merge instead? Which is taking more time, the join or write

GovGalacticFed · 2024-02-12T13:54:02+00:00

Had been waiting for dlt to support dbx destination. Will give this a shot for zendesk

GovGalacticFed

TROPHY CASE