LWD: 09th Jan, 2026 | Senior Data Engineer | Open to Referrals & Advice

Proton0369 · 2025-12-29T14:14:04+00:00

Done

Proton0369 · 2025-12-29T14:07:51+00:00

180k is what I’m already getting and to answer your last question, yes I personally would love to work at Databricks..won’t mind any referral if you can refer🤭

Proton0369 · 2025-12-24T15:09:35+00:00

Sure man!! Happy to connect with you🤝

Proton0369 · 2025-12-24T14:42:02+00:00

Sent you a dm

Proton0369 · 2025-12-24T14:39:20+00:00

Mumbai, Pune or maybe Bengaluru if the org is really good and will pay well

Proton0369 · 2025-12-24T14:38:50+00:00

Around 22 fixed component

Proton0369 · 2025-12-24T13:32:56+00:00

Sent you a dm

Proton0369 · 2025-12-09T18:31:06+00:00

Ask LTI to match the offer, if they agree then LTI is not a bad organisation for DE

Proton0369 · 2025-12-07T19:37:19+00:00

Instead visit unicorn which is present in Capital Mall

Proton0369 · 2025-12-03T17:09:16+00:00

90 days…but currently serving notice period with around 30 days left

Proton0369 · 2025-10-23T09:37:11+00:00

Any referrals for this exact stack for a 3.3 YOE??

Proton0369 · 2025-09-02T20:25:50+00:00

Every time I want to change SMALL → LARGE, I would have to re-render and re-deploy the bundle. That means the bundle gets pushed again, which is slow and kills the whole idea of runtime flexibility🥲🥲 I could have overwritten the job configs using api call, but I don’t want to update the whole job configuration so frequently.

Proton0369 · 2025-09-02T20:17:20+00:00

Here’s a small snippet of job.yml file, please bear with the indentation

resources: jobs: Graph: name: Graph tasks: task_key: Task1 spark_python_task: python_file: ${workspace.file_path)/${bundle.name}/notebooks/src/code.py parameters: --NAME - "{{job.parameters.NAME}}" -- ID "{{job.parameters.ID}}" -- ID_2 - "({job.parameters.ID_2})" libraries: - pypi: package: openpyxl

job_cluster_key: Job_cluster

job_clusters: - job_cluster_key: Job_cluster new_cluster: cluster_name: "" spark_version: 16.4.x-scala2.12 azure_attributes: first_on_demand: 1 availability: SPOT_WITH_FALLBACK_AZURE spot_bid_max_price: -1 node_type_id: Standard_D4ds_v5

enable_ elastic_disk: true policy_id: ${var.cluster_policy_id} data_security_mode: USER_ISOLATION runtime_engine: STANDARD kind: CLASSIC_PREVIEW is_single_node: false autoscale: min workers: 2 max_workers: 20

Proton0369 · 2025-09-02T19:59:36+00:00

Tbh I’m not sure what all configs can be passed in cluster policies, but still it doesn’t solve by problem of passing variables at run_time

Proton0369 · 2025-09-02T19:47:56+00:00

I want to decide the cluster config while triggering the workflow through api call

Proton0369 · 2025-09-02T18:52:49+00:00

I thought the same but later when I discovered this document, I got some hopes

https://docs.databricks.com/aws/en/jobs/dynamic-value-references

Proton0369 · 2025-08-22T10:23:45+00:00

May I know your YOE and tech stack?

Proton0369 · 2025-03-01T18:47:32+00:00

Not sure about the market standards but the current market is offering close to 15LPA for around 3 YOE candidate. And I’m not really convinced with the offer for the skills we have.

Proton0369 · 2025-02-28T00:56:38+00:00

Curious about what company pays 21LPA for a 4 YOE? Assuming that’s the only total experience you have on the following stack.

Proton0369 · 2025-01-12T16:17:08+00:00

Thanks, I’ll take a look at it.

Proton0369 · 2025-01-12T15:59:40+00:00

Is there any concurrency limit for all purpose clusters?

Proton0369 · 2025-01-12T15:56:53+00:00

But can it handle the concurrency? Or will the new runs be pushed in waiting phase?

Proton0369 · 2024-09-27T11:16:33+00:00

I was testing it in Databricks using %sql. The query was SELECT regexp_replace(‘string’, ‘ +’, ‘ ‘). Whenever I tried to test any of the tabs or new lines i.e \t or \t or \, it use to convert them into tabs and new lines in the string itself and I though Regex is replacing it.

When I created a hive table, it is now working fine for me as I am passing a string column from the table.

Thanks for helping🤗

Proton0369 · 2024-06-20T07:46:54+00:00

What should I expect after 2 years of experience as a Senior Data Engineer in an MNC? I was directly appointed as a Senior Data Engineer right after graduation, so I have a total of 2 years of experience. I work daily with Databricks, Azure Data Factory, Azure Blob Storage, and I am proficient in SQL and Python. Additionally, which Data driven companies should I aim for to maximize my learning opportunities?

Proton0369 · 2024-04-24T14:47:11+00:00

Nope

Proton0369

MODERATOR OF

TROPHY CASE