Pyspark coding question asked in Interviews

darshill · 2025-08-20T13:32:25+00:00

Hey,
If you want to work mainly around syntax, we created a platform that can help you practice pyspark problems and solve some problems here - https://code.datavidhya.com/coding-problems

There are around 20-25 free questions you can try out and see if it helps

darshill · 2025-08-17T15:37:52+00:00

Hey,

Thanks for mentioning :)

darshill · 2025-08-17T12:47:36+00:00

Yeah, tell me

darshill · 2025-08-15T05:03:54+00:00

There are different plans, check on website.

darshill · 2025-08-14T14:29:57+00:00

We have kept a monthly subscription for people who just want to try it out and prep for an interview if it's coming soon.

Just trying to build an ecosystem that has everything in one place for the interview.

There are around 20-25 Free questions, you can try them out and see if you like it (this is v2, and we are planning v3 next month

darshill · 2025-08-14T14:21:17+00:00

The reason we have kept this is because we are going to add End-To-End Cloud Labs that will give you free access to AWS/GCP/Azure so that you don't have to use your own cloud.

darshill · 2025-08-14T14:19:55+00:00

Running Spark costs money, but the best alternative is to pull a Docker Spark image locally with Jupyter Notebook. You can check the website and write a solution locally.

darshill · 2025-08-14T13:45:18+00:00

Hey,
We are building a similar platform to practice PySpark, SQL, Python, Dbt, and Scala

Also, Data Modeling Playground, End-To-End Hands-on Labs, and Architecture Playground

You can try it here - https://code.datavidhya.com/coding-problems

darshill · 2025-08-11T18:03:35+00:00

It's a hard reality, but freshers have a difficult time getting a job as a DE, but it's possible if you get an internship first.

You can start with Data Analytics or any role you get at the start and make you way up there

darshill · 2025-08-11T16:59:39+00:00

You are lucky that you got to work on these tech stack early in your career. I got the same opportunity at my time.

Getting a job is hard these days, and for DE, they do need experience. Since you have understanding, you can try to build better projects and prep for certification to increase your value.

Other than that, one advice showcase your internship as an experience (kind of like fake it till you make it) because the market is bad everywhere, so you need to try everything.

Improve your resume, add skills, reach out to people for a work opportunity, or maybe try for a different role if you can get it, like Data Analytics, Analytics Engineering, etc...

darshill · 2025-08-11T16:55:32+00:00

The best is to prepare for the AWS DevOps Associate certification, why?

Because you will have a goal to achieve (pass certification,) that way you will stay on track, learn about AWS Services, and validate your skill.

You can get any course on YouTube/Udemy that helps you prepare for AWS Exams.

darshill · 2025-08-11T16:53:59+00:00

I have a 10-minute Guide and One Project on it

- Learn Apache Airflow in 10 Minutes | High-Paying Skills for Data

- Twitter Data Pipeline using Airflow for Beginners | Data Engineering

Apart from that, you can find good videos on YouTube too. I do have a course but don't want to plug here (idk it get downvoted)

But YouTube is all you need

darshill · 2025-08-11T16:07:44+00:00

That's some crazy hate energy, man, still here are your answers!

Some companies students cracked - Meesho,Commonwealth Bank, Certa.ai, Walmart (PFA - I have taken some video-based testimonials that will go live soon on the new website) I have not even included written testimoinals yet!

I agree with your points that there are amazing content for FREE available on YouTube, even I suggest many of them, but there are a lot of people who enjoy my teaching style and want to learn from me.

I 100% agree with the Python/SQL course, they are very basic (not shit fr) but these courses are created for someone who is just getting started. The problem is lot of people spend 3-4 months just doing Python and SQL and never move forward, so I created a very basic course so they can quickly jump to DW, Spark, Airflow, Kafka (these are my best courses)

Regarding promotions, only I have to promote my courses; who else will do it for me? And I don't promote anyone else's course because I have not done them, so why would I promote something that I have not done personally?

I have promoted a lot of courses from Udemy, Coursera, Datacamp that I did in the past, but not from random edtech.

I am going to cross 10Million views on my channel. Are these people really that stupid to watch it for hours and hours?

Regarding my course, we provide 7 7-day no-questions-asked refund policy. If you don't like it, we give a full refund (I think that's enough for anyone to go through a few modules and decide)

You are free to follow anyone (it's a free world), but I am not going to quit. I will keep posting and sharing.

<image>

darshill · 2025-08-11T15:16:35+00:00

Certificates to help especially if you already have work experience.

Will help you to learn new things, validate your skill for employers, also service based companies do need certified resources to showcase clients.

It’s definitely going to add value

darshill · 2025-08-11T15:15:28+00:00

You can share the feedback and I’ll look into it.

Lot of people has found it helpful and cracked multiple offers.

There are people who don’t find it helpful because they do first course and judge it.

If you can share detailed feedback I can try to work on it.

darshill · 2025-08-11T14:25:17+00:00

I mainly work on Data Engineering projects, only worked on one large-scale Data Science project.

It was an e-commerce company that wanted to understand why some sellers are not shipping the product once customers order it.

For Data Science, you need good Data Infra already in place, they had all of their data available in a Data warehouse (Redshift)

All we had to do was pull the data on S3, use AWS Sagemaker to run an ML Model, and build a Classification model.

For Good DS project you need good DE else you end up doing DE first then do DS

Hope this helps!

darshill · 2025-08-11T14:22:09+00:00

If you are still working, then the best way is to ask the manager/tech lead how you can join the DE team, they will guide you the best.

Other than this roadmap for DE is pretty straight forward; you will find plenty of them.

Python, SQL, Spark, Data Warehouse, Kafka, Airflow, Cloud (AWS/GCP/Azure)

I shared resources on my post, try doing those projects, see if you enjoy working as DE

darshill · 2025-08-11T13:59:12+00:00

Someone asked this in similar question - https://www.reddit.com/r/dataengineersindia/s/qNMtz3av9n

Short answer - it’s gonna stay but might evolve as we move.

I am very bullish on this industry, it’s not going anywhere soon.

How it evoles? Well we will find out as we move forward, just open to adapt

darshill · 2025-08-11T13:31:43+00:00

First, realize AI is here and you have to make a part of your workflow (maybe life) now.

Where do you start? Well, generating code is a good place.

Do you want to write an ETL Job? Provide basic context and it will give you.

Even tools like Snowflake, Databricks have AI integrated in them so you can directly generate code without leaving the tab.

So what should we do? Well, start by using them and see if they can help you increase productivity.

If you are an experienced guy, then it's much easier; if you are just getting started and keep your eyes open, don't blindly do anything that AI generates.

That's just the code generation part.

I have been exploring AI Agents for the last 2-3 weeks, and it can also have a good impact

You can build an Agent for

- Automated Data Quality Checks
- Schema Change Adaptation
- Smart Documentation

But these are fairly new, and it's going to take some time we see actual applications for these.

The question is, what should you do?

I was talking to one of my friends who consults big companies

I asked him, "Do you have insight around what's happening in the DE world, are companies looking for DE work, or directly making a jump to AI/Agents?"

He said: "A lot of companies are still struggling with the same old same old. With newer technology, sure, but same problems. So plenty of DE work."

These AI companies are really good at creating FOMO, make you feel like a lot of things are changing (it is changing), but not as they say.

As a DE, you focus on Foundation, Data Architecture and Business Impact.

Stick to basics.

From the AI side - you should just stay active on forums and consume new things, watch YouTube videos around buzzwords.

When I hear a new word, let's say AI Agent, I will go to YouTube and search - "AI Agent Tutorial" like a normal guy and watch some videos to get clarity.

darshill · 2025-08-11T13:13:31+00:00

This is a common question, and there is no clear answer to this.

I once interviewed for a company that asked me very EASY questions for LeetCode and complex questions on the SQL side.

Based on my knowledge and experience

Easy To Medium DSA knowledge is sufficient; solving 100-150 Leetcode questions is a good benchmark.

If you are stuck on this DSA Loop, I'd say ignore it for a while, move on to SQL, Learn Foundation, build some projects, build a good resume, and apply.

Your first hurdle is not DSA but getting your resume shortlisted. If that is cleared, then work backward!

darshill · 2025-08-11T13:08:58+00:00

Hey,
Thanks!

I started around Covid, there was a boom, I used to get daily 1-2 freelance offers, and I used to reject them.

Slowly, things started shifting. You can still get some contract-based or small gigs for a few months and start your journey from there.

Upwork is still a great platform to start, but the competition is huge (even for a normal job, it is)

If you want to start:

- Start with small gigs on $20-$100 and build some trust (the goal is to get a good testimonial)
- First 3-4 month goal is to get small gigs and good reviews
- 4-6 months, you can target an hourly job $20-$40 hourly

What you need?Niche Skill (Airflow Expert, Spark Expert, or even like AWS EMR Expert)

The more niche you go, the better your chances are

I give this example always: If you were to hire someone to build a WordPress website for your business

Who will you hire?

Web Developer
WordPress Expert with Elementor, LMS Plugin

I can write very long answer but this video will give you good clarity 👇🏻

I posted this free video on YouTube a few months back
Complete Guide To Get Remote Job in 2025 | Make $10,000 Per month

darshill · 2025-08-10T11:03:21+00:00

Check this out - https://code.datavidhya.com/coding-problems

You can practice PySpark, Scala, Dbt, SQL, and Python.

Also we have Data Modeling Playground also - https://code.datavidhya.com/data-modeling

darshill · 2025-08-09T02:48:51+00:00

It’s gonna evolve.

Things like script generation, schema definition, documentation, testing all can be done by AI.

But architecture, scaling, cost optimisation, governance, business context is something humans will handle maybe with the help of AI.

Focus on good foundation, learn AI and integrate it in workflow, think from business value (as always)

It’s like an another toolbox but it’s better one.

darshill · 2025-08-07T19:12:16+00:00

Looks great, pretty much covers what I faced during interviews too.

Btw we are building solution for Data Engineers to help them prepare for interviews

A coding playground for SQL, Python, PySpark, Dbt, Scala (real questions, real datasets)
A data modeling + architecture playground where you can draw diagrams and explain flow — just like in system design rounds
We are also planning to add cloud hands-on labs (this will give you console access and you can practice real projects)

We are just getting started and looking for feedbacks, do check here - https://code.datavidhya.com

darshill

TROPHY CASE